{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:36:29Z","timestamp":1760150189446,"version":"build-2065373602"},"reference-count":28,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,10,20]],"date-time":"2023-10-20T00:00:00Z","timestamp":1697760000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Variational inference provides a way to approximate probability densities through optimization. It does so by optimizing an upper or a lower bound of the likelihood of the observed data (the evidence). The classic variational inference approach suggests maximizing the Evidence Lower Bound (ELBO). Recent studies proposed to optimize the variational R\u00e9nyi bound (VR) and the \u03c7 upper bound. However, these estimates, which are based on the Monte Carlo (MC) approximation, either underestimate the bound or exhibit a high variance. In this work, we introduce a new upper bound, termed the Variational R\u00e9nyi Log Upper bound (VRLU), which is based on the existing VR bound. In contrast to the existing VR bound, the MC approximation of the VRLU bound maintains the upper bound property. Furthermore, we devise a (sandwiched) upper\u2013lower bound variational inference method, termed the Variational R\u00e9nyi Sandwich (VRS), to jointly optimize the upper and lower bounds. We present a set of experiments, designed to evaluate the new VRLU bound and to compare the VRS method with the classic Variational Autoencoder (VAE) and the VR methods. Next, we apply the VRS approximation to the Multiple-Source Adaptation problem (MSA). MSA is a real-world scenario where data are collected from multiple sources that differ from one another by their probability distribution over the input space. The main aim is to combine fairly accurate predictive models from these sources and create an accurate model for new, mixed target domains. However, many domain adaptation methods assume prior knowledge of the data distribution in the source domains. In this work, we apply the suggested VRS density estimate to the Multiple-Source Adaptation problem (MSA) and show, both theoretically and empirically, that it provides tighter error bounds and improved performance, compared to leading MSA methods.<\/jats:p>","DOI":"10.3390\/e25101468","type":"journal-article","created":{"date-parts":[[2023,10,20]],"date-time":"2023-10-20T07:22:32Z","timestamp":1697786552000},"page":"1468","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Variational Inference via R\u00e9nyi Bound Optimization and Multiple-Source Adaptation"],"prefix":"10.3390","volume":"25","author":[{"given":"Dana","family":"Zalman (Oshri)","sequence":"first","affiliation":[{"name":"School of Computer Science, Reichman University, Herzliya 4610101, Israel"},{"name":"Data Science Institute, Reichman University, Herzliya 4610101, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9346-8774","authenticated-orcid":false,"given":"Shai","family":"Fine","sequence":"additional","affiliation":[{"name":"Data Science Institute, Reichman University, Herzliya 4610101, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1023\/A:1007665907178","article-title":"An introduction to variational methods for graphical models","volume":"37","author":"Jordan","year":"1999","journal-title":"Mach. Learn."},{"key":"ref_2","first-page":"1","article-title":"Auto-Encoding Variational Bayes","volume":"1050","author":"Kingma","year":"2014","journal-title":"Statistics"},{"key":"ref_3","unstructured":"Li, Y., and Turner, R.E. (2016). R\u00e9nyi divergence variational inference. Adv. Neural Inf. Process. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2008","DOI":"10.1109\/TPAMI.2018.2889774","article-title":"Advances in variational inference","volume":"41","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","first-page":"1303","article-title":"Stochastic variational inference","volume":"14","author":"Hoffman","year":"2013","journal-title":"J. Mach. Learn. Res."},{"key":"ref_6","first-page":"17370","article-title":"F-divergence variational inference","volume":"33","author":"Wan","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_7","unstructured":"Akrami, H., Joshi, A.A., Li, J., Aydore, S., and Leahy, R.M. (2019). Robust Variational Autoencoder. arXiv."},{"key":"ref_8","unstructured":"Hernandez-Lobato, J., Li, Y., Rowland, M., Bui, T., Hern\u00e1ndez-Lobato, D., and Turner, R. (2016, January 19\u201324). Black-box alpha divergence minimization. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_9","unstructured":"Hoffman, J., Mohri, M., and Zhang, N. (2018). Algorithms and theory for multiple-source adaptation. Adv. Neural Inf. Process. Syst."},{"key":"ref_10","first-page":"146","article-title":"I-divergence geometry of probability distributions and minimization problems","volume":"3","year":"1975","journal-title":"Ann. Probab."},{"key":"ref_11","unstructured":"MacKay, D.J. (2003). Information Theory, Inference and Learning Algorithms, Cambridge University Press."},{"key":"ref_12","unstructured":"Shekhovtsov, A., Schlesinger, D., and Flach, B. (2021, January 4). VAE Approximation Error: ELBO and Exponential Families. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1109\/TIT.2014.2320500","article-title":"R\u00e9nyi divergence and Kullback-Leibler divergence","volume":"60","author":"Harremos","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_14","unstructured":"Dieng, A.B., Tran, D., Ranganath, R., Paisley, J., and Blei, D. (2017). Variational Inference via \u03c7 Upper Bound Minimization. Adv. Neural Inf. Process. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1002\/wics.1314","article-title":"Why the Monte Carlo method is so important today","volume":"6","author":"Kroese","year":"2014","journal-title":"Wiley Interdiscip. Rev. Comput. Stat."},{"key":"ref_16","unstructured":"Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. (2019, January 9\u201315). On variational bounds of mutual information. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_17","unstructured":"Ji, C., and Shen, H. (2019). Stochastic variational inference via upper bound. arXiv."},{"key":"ref_18","unstructured":"Pradier, M.F., Hughes, M.C., and Doshi-Velez, F. (2023, September 10). Challenges in Computing and Optimizing Upper Bounds of Marginal Likelihood Based on Chi-Square Divergences. Available online: https:\/\/openreview.net\/forum?id=BJxk51h4FS."},{"key":"ref_19","first-page":"1041","article-title":"Domain adaptation with multiple sources","volume":"21","author":"Mansour","year":"2008","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_20","unstructured":"Cortes, C., Mohri, M., Suresh, A.T., and Zhang, N. (2021, January 18\u201324). A discriminative technique for multiple-source adaptation. Proceedings of the International Conference on Machine Learning, Virtual Event."},{"key":"ref_21","unstructured":"Mansour, Y., Mohri, M., and Rostamizadeh, A. (2009, January 18\u201321). Multiple source adaptation and the R\u00e9nyi divergence. Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada."},{"key":"ref_22","first-page":"7","article-title":"Generalizations of H\u00f6lders inequality","volume":"26","author":"Cheung","year":"2001","journal-title":"Sciences"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1023\/A:1021765131316","article-title":"DC programming: Overview","volume":"103","author":"Horst","year":"1999","journal-title":"J. Optim. Theory Appl."},{"key":"ref_24","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_25","unstructured":"French, G., Mackiewicz, M., and Fisher, M. (May, January 30). Self-ensembling for visual domain adaptation. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Saenko, K., Kulis, B., Fritz, M., and Darrell, T. (2010, January 5\u201311). Adapting visual category models to new domains. Proceedings of the European conference on computer vision, Heraklion, Greece.","DOI":"10.1007\/978-3-642-15561-1_16"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1561\/2200000056","article-title":"An introduction to variational autoencoders","volume":"12","author":"Kingma","year":"2019","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3555","DOI":"10.1109\/TMI.2021.3090412","article-title":"Unsupervised domain adaptation with variational approximation for cardiac segmentation","volume":"40","author":"Wu","year":"2021","journal-title":"IEEE Trans. Med Imaging"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1468\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:10:18Z","timestamp":1760130618000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/10\/1468"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,20]]},"references-count":28,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["e25101468"],"URL":"https:\/\/doi.org\/10.3390\/e25101468","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2023,10,20]]}}}