{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,11]],"date-time":"2026-10-11T14:36:48Z","timestamp":1791729408790,"version":"4.3.4"},"reference-count":33,"publisher":"Institute of Mathematical Statistics","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bayesian Anal."],"published-print":{"date-parts":[[2025,6,1]]},"abstract":"Simulation-based calibration checking (SBC) is a practical method to validate computationally-derived posterior distributions or their approximations. In this paper, we introduce a new variant of SBC to alleviate several known problems. Our variant allows the user to in principle detect any possible issue with the posterior, while previously reported implementations could never detect large classes of problems including when the posterior is equal to the prior. This is made possible by including additional data-dependent test quantities when running SBC. We argue and demonstrate that the joint likelihood of the data is an especially useful test quantity. Some other types of test quantities and their theoretical and practical benefits are also investigated. We provide theoretical analysis of SBC, thereby providing a more complete understanding of the underlying statistical mechanisms. We also bring attention to a relatively common mistake in the literature and clarify the difference between SBC and checks based on the data-averaged posterior. We support our recommendations with numerical case studies on a multivariate normal example and a case study in implementing an ordered simplex data type for use with Hamiltonian Monte Carlo. The SBC variant introduced in this paper is implemented in the SBC R package.<\/jats:p>","DOI":"10.1214\/23-ba1404","type":"journal-article","created":{"date-parts":[[2023,11,23]],"date-time":"2023-11-23T07:14:17Z","timestamp":1700723657000},"page":"461-488","source":"Crossref","is-referenced-by-count":29,"title":["Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity"],"prefix":"10.1214","volume":"20","author":[{"given":"Martin","family":"Modr\u00e1k","sequence":"first","affiliation":[{"name":"Institute of Microbiology of the Czech Academy of Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angie H.","family":"Moon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shinyoung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"B\u00fcrkner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niko","family":"Huurre","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kate\u0159ina","family":"Faltejskov\u00e1","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Gelman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aki","family":"Vehtari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"9","unstructured":"Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., B\u00fcrkner, P.-C., and Modr\u00e1k, M. (2020). \u201cBayesian workflow.\u201d arXiv:2011.01808<\/a>."},{"key":"4","doi-asserted-by":"crossref","unstructured":"Cook, S. R., Gelman, A., and Rubin, D. B. (2006). \u201cValidation of Software for Bayesian Models Using Posterior Quantiles.\u201d Journal of Computational and Graphical Statistics<\/i>, 15(3): 675\u2013692.","DOI":"10.1198\/106186006X136976"},{"key":"1","doi-asserted-by":"publisher","unstructured":"Aguilar, J. E. and B\u00fcrkner, P.-C. (2023). \u201cIntuitive joint priors for Bayesian linear multilevel models: The R2D2M2 prior.\u201d Electronic Journal of Statistics<\/i>, 17(1): 1711\u20131767.","DOI":"10.1214\/23-ejs2136"},{"key":"2","doi-asserted-by":"crossref","unstructured":"Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017). \u201cStan: A probabilistic programming language.\u201d Journal of Statistical Software<\/i>, 76(1). URL https:\/\/www.jstatsoft.org\/index.php\/jss\/article\/view\/v076i01<\/a>","DOI":"10.18637\/jss.v076.i01"},{"key":"3","unstructured":"Cockayne, J., Graham, M. M., Oates, C. J., Sullivan, T. J., and Teymur, O. (2022). \u201cTesting whether a learning procedure is calibrated.\u201d Journal of Machine Learning Research<\/i>, 23(203): 1\u201336. URL http:\/\/jmlr.org\/papers\/v23\/21-1065.html<\/a>"},{"key":"5","unstructured":"Cusumano-Towner, M. F. and Mansinghka, V. K. (2017). \u201cAIDE: An algorithm for measuring the accuracy of probabilistic inference algorithms.\u201d In Proceedings of the 31st International Conference on Neural Information Processing Systems<\/i>, NIPS\u201917, 3004\u20133014. Red Hook, NY, USA: Curran Associates Inc."},{"key":"6","unstructured":"Domke, J. (2021). \u201cAn easy to interpret diagnostic for approximate inference: Symmetric divergence over simulations.\u201d arXiv:2103.01030<\/a>"},{"key":"7","doi-asserted-by":"publisher","unstructured":"Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., and Gelman, A. (2019). \u201cVisualization in Bayesian workflow.\u201d Journal of the Royal Statistical Society: Series A<\/i>, 182: 389\u2013402.","DOI":"10.1111\/rssa.12378"},{"key":"8","unstructured":"Gandy, A. and Scott, J. (2020). \u201cUnit testing for MCMC and other Monte Carlo methods.\u201d arXiv:2001.06465<\/a>"},{"key":"10","doi-asserted-by":"publisher","unstructured":"Geweke, J. (2004). \u201cGetting it right.\u201d Journal of the American Statistical Association<\/i>, 99: 799\u2013804.","DOI":"10.1198\/016214504000001132"},{"key":"11","doi-asserted-by":"publisher","unstructured":"Grinsztajn, L., Semenova, E., Margossian, C. C., and Riou, J. (2021). \u201cBayesian workflow for disease transmission modeling in Stan.\u201d Statistics in Medicine<\/i>, 40: 6209\u20136234.","DOI":"10.1002\/sim.9164"},{"key":"12","unstructured":"Grosse, R. B., Ancha, S., and Roy, D. M. (2016). \u201cMeasuring the reliability of MCMC inference with bidirectional Monte Carlo.\u201d In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems<\/i>, volume 29. Curran Associates. URL https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2016\/file\/0e9fa1f3e9e66792401a6972d477dcc3-Paper.pdf<\/a>"},{"key":"13","unstructured":"Kay, M. (2021). \u201cExtracting and visualizing tidy residuals from Bayesian models.\u201d URL http:\/\/mjskay.github.io\/tidybayes\/articles\/tidybayes-residuals.html<\/a>"},{"key":"14","unstructured":"Kim, S., Moon, A. H., Modr\u00e1k, M., and S\u00e4ilynoja, T. (2022). \u201cSBC: Simulation based calibration for rstan\/cmdstanr models.\u201d URL https:\/\/github.com\/hyunjimoon\/SBC\/<\/a>"},{"key":"15","doi-asserted-by":"publisher","unstructured":"Lee, J. E., Nicholls, G. K., and Ryder, R. J. (2019). \u201cCalibration procedures for approximate Bayesian credible sets.\u201d Bayesian Analysis<\/i>, 14: 1245\u20131269.","DOI":"10.1214\/19-BA1175"},{"key":"16","unstructured":"Lueckmann, J.-M., Boelts, J., Greenberg, D., Goncalves, P., and Macke, J. (2021). \u201cBenchmarking simulation-based inference.\u201d Proceedings of Machine Learning Research<\/i>, 130: 343\u2013351. URL https:\/\/proceedings.mlr.press\/v130\/lueckmann21a.html<\/a>"},{"key":"17","doi-asserted-by":"crossref","unstructured":"Mcleod, J. and Simpson, F. (2021). \u201cValidating Gaussian process models with simulation-based calibration.\u201d In 2021 IEEE International Conference on Artificial Intelligence Testing (AITest)<\/i>, 101\u2013102.","DOI":"10.1109\/AITEST52744.2021.00028"},{"key":"18","unstructured":"Modr\u00e1k, M., Moon, A. H., Kim, S., B\u00fcrkner, P., Huurre, N., Faltejskov\u00e1, K., Gelman, A., and Vehtari, A. (2023). \u201cSupplementary Material for \u201cSimulation-based calibration checking for Bayesian computation: The choice of test quantities shapes sensitivity\u201d.\u201d"},{"key":"19","doi-asserted-by":"publisher","unstructured":"Prangle, D., Blum, M. G. B., Popovic, G., and Sisson, S. A. (2014). \u201cDiagnostic tools for approximate Bayesian computation using the coverage property.\u201d Australian & New Zealand Journal of Statistics<\/i>, 56: 309\u2013329.","DOI":"10.1111\/anzs.12087"},{"key":"20","doi-asserted-by":"publisher","unstructured":"Radev, S. T., D\u2019Alessandro, M., Mertens, U. K., Voss, A., K\u00f6the, U., and B\u00fcrkner, P.-C. (2021). \u201cAmortized Bayesian model comparison with evidential deep learning.\u201d IEEE Transactions on Neural Networks and Learning Systems<\/i>, 1\u201315.","DOI":"10.1109\/tnnls.2017.2665555"},{"key":"21","doi-asserted-by":"crossref","unstructured":"Radev, S. T., Mertens, U. K., Voss, A., Ardizzone, L., and K\u00f6the, U. (2020). \u201cBayesFlow: Learning complex stochastic models with invertible neural networks.\u201d IEEE Transactions on Neural Networks and Learning Systems<\/i>, 33(4): 1452\u20131466.","DOI":"10.1109\/TNNLS.2020.3042395"},{"key":"22","unstructured":"Radev, S. T., Schmitt, M., Pratz, V., Picchini, U., K\u00f6the, U., and B\u00fcrkner, P.-C. (2023). \u201cJANA: Jointly amortized neural approximation of complex Bayesian models.\u201d In Uncertainty in Artificial Intelligence (UAI) Conference Proceedings<\/i>."},{"key":"23","unstructured":"Ramesh, P., Lueckmann, J.-M., Boelts, J., Tejero-Cantero, \u00c1., Greenberg, D. S., Goncalves, P. J., and Macke, J. H. (2022). \u201cGATSBI: Generative Adversarial Training for Simulation-Based Inference.\u201d In International Conference on Learning Representations<\/i>. URL https:\/\/openreview.net\/forum?id=kR1hC6j48Tp<\/a>"},{"key":"24","unstructured":"Rendsburg, L., Kristiadi, A., Hennig, P., and Von Luxburg, U. (2022). \u201cDiscovering inductive bias with Gibbs priors: A diagnostic tool for approximate Bayesian inference.\u201d Proceedings of Machine Learning Research<\/i>, 151: 1503\u20131526. URL https:\/\/proceedings.mlr.press\/v151\/rendsburg22a.html<\/a>"},{"key":"25","unstructured":"Saad, F. A., Freer, C. E., Ackerman, N. L., and Mansinghka, V. K. (2019). \u201cA family of exact goodness-of-fit tests for high-dimensional discrete distributions.\u201d Proceedings of Machine Learning Research<\/i>, 89: 1640\u20131649. URL https:\/\/proceedings.mlr.press\/v89\/saad19a.html<\/a>"},{"key":"26","doi-asserted-by":"publisher","unstructured":"S\u00e4ilynoja, T., B\u00fcrkner, P.-C., and Vehtari, A. (2022). \u201cGraphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison.\u201d Statistics and Computing<\/i>, 32(2).","DOI":"10.1007\/s11222-022-10090-6"},{"key":"27","doi-asserted-by":"crossref","unstructured":"Schad, D. J., Nicenboim, B., B\u00fcrkner, P.-C., Betancourt, M., and Vasishth, S. (2022). \u201cWorkflow techniques for the robust use of Bayes factors.\u201d Psychological Methods<\/i>. URL https:\/\/doi.org\/10.1037\/met0000472<\/a>","DOI":"10.1037\/met0000472"},{"key":"28","unstructured":"Talts, S., Betancourt, M., Simpson, D., Vehtari, A., and Gelman, A. (2020). \u201cValidating Bayesian inference algorithms with simulation-based calibration.\u201d URL http:\/\/www.stat.columbia.edu\/~gelman\/research\/unpublished\/sbc.pdf<\/a>"},{"key":"29","doi-asserted-by":"publisher","unstructured":"Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., and B\u00fcrkner, P.-C. (2021). \u201cRank-normalization, folding, and localization: An improved R<\/mi><\/mrow>\u02c6<\/mo><\/mover><\/math> for assessing convergence of MCMC (with discussion).\u201d Bayesian Analysis<\/i>, 16(2): 667\u2013718.","DOI":"10.1214\/20-ba1221"},{"key":"30","unstructured":"Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). \u201cYes, but did it work?: Evaluating variational inference.\u201d Proceedings of Machine Learning Research<\/i>, 80: 5581\u20135590. URL https:\/\/proceedings.mlr.press\/v80\/yao18a.html<\/a>"},{"key":"31","doi-asserted-by":"publisher","unstructured":"Yu, X., Nott, D. J., Tran, M.-N., and Klein, N. (2021). \u201cAssessment and adjustment of approximate inference algorithms using the law of total variance.\u201d Journal of Computational and Graphical Statistics<\/i>, 30: 977\u2013990.","DOI":"10.1080\/10618600.2021.1880921"},{"key":"32","doi-asserted-by":"publisher","unstructured":"Zhang, Y. D., Naughton, B. P., Bondell, H. D., and Reich, B. J. (2020). \u201cBayesian regression using a prior on the model fit: The R2-D2 shrinkage prior.\u201d Journal of the American Statistical Association<\/i>, 117: 862\u2013874.","DOI":"10.1080\/01621459.2020.1825449"},{"key":"33","unstructured":"Zhao, D., Dalmasso, N., Izbicki, R., and Lee, A. B. (2021). \u201cDiagnostics for conditional density models and Bayesian inference algorithms.\u201d Proceedings of Machine Learning Research<\/i>, 161: 1830\u20131840. URL https:\/\/proceedings.mlr.press\/v161\/zhao21b.html<\/a>"}],"container-title":["Bayesian Analysis"],"original-title":[],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:41:08Z","timestamp":1791391268000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/bayesian-analysis\/volume-20\/issue-2\/Simulation-Based-Calibration-Checking-for-Bayesian-Computation--The-Choice\/10.1214\/23-BA1404.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,1]]},"references-count":33,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,6,1]]}},"URL":"https:\/\/doi.org\/10.1214\/23-ba1404","relation":{},"ISSN":["1936-0975","1931-6690"],"issn-type":[{"value":"1936-0975","type":"print"},{"value":"1931-6690","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,1]]}}}