{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T16:46:09Z","timestamp":1780764369593,"version":"3.54.1"},"reference-count":77,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,6,24]],"date-time":"2021-06-24T00:00:00Z","timestamp":1624492800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Accurate evaluation of Bayesian model evidence for a given data set is a fundamental problem in model development. Since evidence evaluations are usually intractable, in practice variational free energy (VFE) minimization provides an attractive alternative, as the VFE is an upper bound on negative model log-evidence (NLE). In order to improve tractability of the VFE, it is common to manipulate the constraints in the search space for the posterior distribution of the latent variables. Unfortunately, constraint manipulation may also lead to a less accurate estimate of the NLE. Thus, constraint manipulation implies an engineering trade-off between tractability and accuracy of model evidence estimation. In this paper, we develop a unifying account of constraint manipulation for variational inference in models that can be represented by a (Forney-style) factor graph, for which we identify the Bethe Free Energy as an approximation to the VFE. We derive well-known message passing algorithms from first principles, as the result of minimizing the constrained Bethe Free Energy (BFE). The proposed method supports evaluation of the BFE in factor graphs for model scoring and development of new message passing-based inference algorithms that potentially improve evidence estimation accuracy.<\/jats:p>","DOI":"10.3390\/e23070807","type":"journal-article","created":{"date-parts":[[2021,6,24]],"date-time":"2021-06-24T23:22:14Z","timestamp":1624576934000},"page":"807","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Variational Message Passing and Local Constraint Manipulation in Factor Graphs"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7355-2138","authenticated-orcid":false,"given":"\u0130smail","family":"\u015een\u00f6z","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thijs","family":"van de Laar","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dmitry","family":"Bagaev","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bert de","family":"de Vries","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"},{"name":"GN Hearing, JF Kennedylaan 2, 5612 AB Eindhoven, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1146\/annurev-statistics-022513-115657","article-title":"Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models","volume":"1","author":"Blei","year":"2014","journal-title":"Annu. Rev. Stat. Appl."},{"key":"ref_2","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On Information and Sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1109\/18.910573","article-title":"Codes on graphs: Normal realizations","volume":"47","author":"Forney","year":"2001","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MSP.2004.1267047","article-title":"An introduction to factor graphs","volume":"21","author":"Loeliger","year":"2004","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_6","first-page":"661","article-title":"Variational message passing","volume":"6","author":"Winn","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","unstructured":"Yedidia, J.S., Freeman, W.T., and Weiss, Y. (2001). Understanding Belief Propagation and Its Generalizations, Mitsubishi Electric Research Laboratories, Inc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.ijar.2018.11.002","article-title":"A factor graph approach to automated design of Bayesian signal processing algorithms","volume":"104","author":"Cox","year":"2019","journal-title":"Int. J. Approx. Reason."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yedidia, J.S. (2000). An Idiosyncratic Journey beyond Mean Field Theory. Advanced Mean Field Methods, The MIT Press.","DOI":"10.7551\/mitpress\/1100.003.0007"},{"key":"ref_10","unstructured":"Yedidia, J.S., Freeman, W.T., and Weiss, Y. (2001). Bethe Free Energy, Kikuchi Approximations, and Belief Propagation Algorithms, Mitsubishi Electric Research Laboratories, Inc."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Dauwels, J. (2007, January 24\u201329). On Variational Message Passing on Factor Graphs. Proceedings of the IEEE International Symposium on Information Theory, Nice, France.","DOI":"10.1109\/ISIT.2007.4557602"},{"key":"ref_12","unstructured":"Zhang, D., Wang, W., Fettweis, G., and Gao, X. (2017). Unifying Message Passing Algorithms under the Framework of Constrained Bethe Free Energy Minimization. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"van de Laar, T., \u015een\u00f6z, I., \u00d6z\u00e7elikkale, A., and Wymeersch, H. (2021). Chance-Constrained Active Inference. arXiv.","DOI":"10.1162\/neco_a_01427"},{"key":"ref_14","unstructured":"Smola, A.J., Vishwanathan, S.V.N., and Eskin, E. (2004). Laplace propagation. NIPS, The MIT Press."},{"key":"ref_15","unstructured":"Minka, T. (2021, June 24). Divergence Measures and Message Passing. Available online: https:\/\/www.microsoft.com\/en-us\/research\/publication\/divergence-measures-and-message-passing\/."},{"key":"ref_16","unstructured":"Yedidia, J.S. (2021, June 24). Generalized Belief Propagation and Free Energy Minimization. Available online: http:\/\/cba.mit.edu\/events\/03.11.ASE\/docs\/Yedidia.pdf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2282","DOI":"10.1109\/TIT.2005.850085","article-title":"Constructing free-energy approximations and generalized belief propagation algorithms","volume":"51","author":"Yedidia","year":"2005","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_18","unstructured":"Minka, T.P. (2001, January 2\u20135). Expectation Propagation for Approximate Bayesian Inference. Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, Seattle, WA, USA."},{"key":"ref_19","unstructured":"Heskes, T. (2003). Stable fixed points of loopy belief propagation are local minima of the bethe free energy. Advances in Neural Information Processing Systems, The MIT Press."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1109\/18.910572","article-title":"Factor graphs and the sum-product algorithm","volume":"47","author":"Kschischang","year":"2001","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_21","unstructured":"Hoffman, M., Blei, D.M., Wang, C., and Paisley, J. (2012). Stochastic Variational Inference. arXiv."},{"key":"ref_22","unstructured":"Archer, E., Park, I.M., Buesing, L., Cunningham, J., and Paninski, L. (2015). Black box variational inference for state space models. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann Publishers Inc.","DOI":"10.1016\/B978-0-08-051489-5.50008-4"},{"key":"ref_24","first-page":"1","article-title":"Graphical Models, Exponential Families, and Variational Inference","volume":"1","author":"Wainwright","year":"2008","journal-title":"Found. Trends\u00ae Mach. Learn."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chertkov, M., and Chernyak, V.Y. (2006). Loop Calculus in Statistical Physics and Information Science. Phys. Rev. E, 73.","DOI":"10.1103\/PhysRevE.73.065102"},{"key":"ref_26","unstructured":"Weller, A., Tang, K., Jebara, T., and Sontag, D.A. (2014, January 23\u201327). Understanding the Bethe approximation: When and how can it go wrong?. Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, Quebec City, QC, Canada."},{"key":"ref_27","unstructured":"Sibel, J.C. (2021, June 24). Region-Based Approximation to Solve Inference in Loopy Factor Graphs: Decoding LDPC Codes by Generalized Belief Propagation. Available online: https:\/\/hal.archives-ouvertes.fr\/tel-00905668."},{"key":"ref_28","unstructured":"Minka, T. (1999). From Hidden Markov Models to Linear Dynamical Systems, VIsion and Modeling Group, Media Lab, MIT. Technical Report 531."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1109\/JPROC.2007.896497","article-title":"The Factor Graph Approach to Model-Based Signal Processing","volume":"95","author":"Loeliger","year":"2007","journal-title":"Proc. IEEE"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Loeliger, H.A., Bolliger, L., Reller, C., and Korl, S. (2009, January 8\u201313). Localizing, forgetting, and likelihood filtering in state-space models. Proceedings of the 2009 Information Theory and Applications Workshop, La Jolla, CA, USA.","DOI":"10.1109\/ITA.2009.5044943"},{"key":"ref_31","unstructured":"Korl, S. (2005). A Factor Graph Approach to Signal Modelling, System Identification and Filtering. [Ph.D. Thesis, Swiss Federal Institute of Technology]."},{"key":"ref_32","unstructured":"Pearl, J. (1982, January 18\u201320). Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach. Proceedings of the Second AAAI Conference on Artificial Intelligence, Pittsburgh, PA, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1613\/jair.1933","article-title":"Convexity arguments for efficient minimization of the Bethe and Kikuchi free energies","volume":"26","author":"Heskes","year":"2006","journal-title":"J. Artif. Intell. Res."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"S\u00e4rkk\u00e4, S. (2013). Bayesian Filtering and Smoothing, Cambridge University Press.","DOI":"10.1017\/CBO9781139344203"},{"key":"ref_35","unstructured":"Khan, M.E., and Lin, W. (2017). Conjugate-Computation Variational Inference: Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models. arXiv."},{"key":"ref_36","unstructured":"Logan, B., and Moreno, P. (1998, January 15). Factorial HMMs for acoustic modeling. Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, Seattle, WA, USA."},{"key":"ref_37","unstructured":"Hoffman, M.D., and Blei, D.M. (2014). Structured Stochastic Variational Inference. arXiv."},{"key":"ref_38","unstructured":"Singh, R., Ling, J., and Doshi-Velez, F. (2017, January 4\u20139). Structured Variational Autoencoders for the Beta-Bernoulli Process. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_39","unstructured":"Bamler, R., and Mandt, S. (2017). Structured Black Box Variational Inference for Latent Time Series Models. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1016\/j.sigpro.2016.08.027","article-title":"Low Complexity Sparse Bayesian Learning Using Combined BP and MF with a Stretched Factor Graph","volume":"131","author":"Zhang","year":"2017","journal-title":"Signal Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1080\/01621459.2016.1197833","article-title":"Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing","volume":"112","author":"Wand","year":"2017","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_42","unstructured":"Caticha, A. (2012, January 18\u201322). Entropic Inference and the Foundations of Physics. Proceedings of the 11th Brazilian Meeting on Bayesian Statistics, Amparo, Brazil."},{"key":"ref_43","unstructured":"Pearl, J. (2021, June 24). A Probabilistic Calculus of Actions. Available online: https:\/\/arxiv.org\/ftp\/arxiv\/papers\/1302\/1302.6835.pdf."},{"key":"ref_44","unstructured":"Zoeter, O., and Heskes, T. (2005, January 6\u20138). Gaussian Quadrature Based Expectation Propagation. Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Bridgetown, Barbados."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/TAC.2009.2019800","article-title":"Cubature Kalman Filters","volume":"54","author":"Arasaratnam","year":"2009","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_46","unstructured":"Sarkka, S. (2021, June 24). Bayesian Estimation of Time-Varying Systems: Discrete-Time Systems. Available online: https:\/\/users.aalto.fi\/~ssarkka\/course_k2011\/pdf\/course_booklet_2011.pdf."},{"key":"ref_47","unstructured":"Gelman, A., Vehtari, A., Jyl\u00e4nki, P., Robert, C., Chopin, N., and Cunningham, J.P. (2014). Expectation propagation as a way of life. arXiv."},{"key":"ref_48","unstructured":"Deisenroth, M.P., and Mohamed, S. (2012). Expectation Propagation in Gaussian Process Dynamical Systems: Extended Version. arXiv."},{"key":"ref_49","unstructured":"Teh, Y.W., Hasenclever, L., Lienart, T., Vollmer, S., Webb, S., Lakshminarayanan, B., and Blundell, C. (2015). Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Rasmussen, C.E., and Williams, C.K.I. (2006). Gaussian Processes for Machine Learning, MIT Press.","DOI":"10.7551\/mitpress\/3206.001.0001"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Cox, M. (2018, January 3\u20137). Robust Expectation Propagation in Factor Graphs Involving Both Continuous and Binary Variables. Proceedings of the 26th European Signal Processing Conference (EUSIPCO), Rome, Italy.","DOI":"10.23919\/EUSIPCO.2018.8553490"},{"key":"ref_52","unstructured":"Minka, T., Winn, J., Guiver, J., Webster, S., Zaykov, Y., Yangel, B., Spengler, A., and Bronskill, J. (2021, June 23). Infer.NET 2.6. Available online: http:\/\/research.microsoft.com\/infernet."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1273","DOI":"10.1016\/S1053-8119(03)00202-7","article-title":"Dynamic causal modelling","volume":"19","author":"Friston","year":"2003","journal-title":"Neuroimage"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Mathys, C.D., Daunizeau, J., Friston, K.J., and Klaas, S.E. (2011). A Bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci., 5.","DOI":"10.3389\/fnhum.2011.00039"},{"key":"ref_55","first-page":"70","article-title":"A free energy principle for the brain","volume":"100","author":"Friston","year":"2006","journal-title":"J. Physiol."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1016\/j.tics.2009.04.005","article-title":"The free-energy principle: A rough guide to the brain?","volume":"13","author":"Friston","year":"2009","journal-title":"Trends Cogn. Sci."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum Likelihood from Incomplete Data via the EM Algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"J. R. Stat. Soc. Ser. B Methodol."},{"key":"ref_58","unstructured":"Dauwels, J., Eckford, A., Korl, S., and Loeliger, H.A. (2009). Expectation maximization as message passing\u2014Part I: Principles and gaussian messages. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"2215","DOI":"10.1016\/j.physa.2011.02.020","article-title":"Entropy and complexity analysis of Dirac-delta-like quantum potentials","volume":"390","author":"Bouvrie","year":"2011","journal-title":"Phys. A Stat. Mech. Appl."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Dauwels, J., Korl, S., and Loeliger, H.A. (2005, January 4\u20139). Expectation maximization as message passing. Proceedings of the International Symposium on Information Theory 2005, (ISIT 2005), Adelaide, Australia.","DOI":"10.1109\/ISIT.2005.1523402"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Cox, M., van de Laar, T., and de Vries, B. (2018, January 4\u20136). ForneyLab.jl: Fast and flexible automated inference through message passing in Julia. Proceedings of the International Conference on Probabilistic Programming, Boston, MA, USA.","DOI":"10.3389\/frobt.2019.00020"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1137\/141000671","article-title":"Julia: A Fresh Approach to Numerical Computing","volume":"59","author":"Bezanson","year":"2017","journal-title":"SIAM Rev."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"\u015een\u00f6z, I., and de Vries, B. (2018, January 17\u201320). Online Variational Message Passing in the Hierarchical Gaussian Filter. Proceedings of the 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), Aalborg, Denmark.","DOI":"10.1109\/MLSP.2018.8517019"},{"key":"ref_64","unstructured":"Mathys, C.D. (2014). Uncertainty, Precision, and Prediction Errors, UCL. UCL Computational Psychiatry Course."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"\u015een\u00f6z, I., and de Vries, B. (2020, January 21\u201326). Online Message Passing-based Inference in the Hierarchical Gaussian Filter. Proceedings of the 2020 IEEE International Symposium on Information Theory (ISIT), Los Angeles, CA, USA.","DOI":"10.1109\/ISIT44484.2020.9173980"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Podusenko, A., Kouw, W.M., and de Vries, B. (2020, January 21\u201326). Online Variational Message Passing in Hierarchical Autoregressive Models. Proceedings of the 2020 IEEE International Symposium on Information Theory (ISIT), Los Angeles, CA, USA.","DOI":"10.1109\/ISIT44484.2020.9174134"},{"key":"ref_67","unstructured":"Welling, M. (2012). On the Choice of Regions for Generalized Belief Propagation. arXiv."},{"key":"ref_68","unstructured":"Welling, M., Minka, T.P., and Teh, Y.W. (2012). Structured Region Graphs: Morphing EP into GBP. arXiv."},{"key":"ref_69","unstructured":"Loeliger, H.A. (2019, April 03). Factor Graphs and Message Passing Algorithms\u2014Part 1: Introduction. Available online: http:\/\/www.crm.sns.it\/media\/course\/1524\/Loeliger_A.pdf."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1063\/1.1751358","article-title":"Relative Entropy and Inductive Inference","volume":"707","author":"Caticha","year":"2004","journal-title":"AIP Conf. Proc."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"475","DOI":"10.1613\/jair.3062","article-title":"A Minimum Relative Entropy Principle for Learning and Acting","volume":"38","author":"Ortega","year":"2010","journal-title":"J. Artif. Intell. Res."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/TIT.1980.1056144","article-title":"Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy","volume":"26","author":"Shore","year":"1980","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Engel, E., and Dreizler, R.M. (2011). Density Functional Theory: An Advanced Course, Springer. Theoretical and Mathematical Physics.","DOI":"10.1007\/978-3-642-14090-7"},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Boyd, S.P., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.","DOI":"10.1017\/CBO9780511804441"},{"key":"ref_75","unstructured":"Lanczos, C. (2012). The Variational Principles of Mechanics, Courier Corporation."},{"key":"ref_76","unstructured":"Ahn, S., Chertkov, M., and Shin, J. (2021, June 24). Gauging Variational Inference. Available online: https:\/\/dl.acm.org\/doi\/10.5555\/3294996.3295048."},{"key":"ref_77","unstructured":"Tran, V.H. (2018). Copula Variational Bayes inference via information geometry. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/7\/807\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:23:33Z","timestamp":1760163813000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/7\/807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,24]]},"references-count":77,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["e23070807"],"URL":"https:\/\/doi.org\/10.3390\/e23070807","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,24]]}}}