{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T16:47:35Z","timestamp":1791910055649,"version":"4.3.4"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T00:00:00Z","timestamp":1694736000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000893","name":"Simons Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000893","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000879","name":"Sloan Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100014895","name":"Open Philanthropy","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100014895","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"NHLBI","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Eric and Wendy Schmidt Center"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Bayesian modelling helps applied researchers to articulate assumptions about their data and develop models tailored for specific applications. Thanks to good methods for approximate posterior inference, researchers can now easily build, use, and revise complicated Bayesian models for large and rich data. These capabilities, however, bring into focus the problem of model criticism. Researchers need tools to diagnose the fitness of their models, to understand where they fall short, and to guide their revision. In this paper, we develop a new method for Bayesian model criticism, the holdout predictive check (HPC). Holdout predictive check are built on posterior predictive check (PPC), a seminal method that checks a model by assessing the posterior predictive distribution on the observed data. However, PPC use the data twice\u2014both to calculate the posterior predictive and to evaluate it\u2014which can lead to uncalibrated p-values. Holdout predictive check, in contrast, compare the posterior predictive distribution to a draw from the population distribution, a heldout dataset. This method blends Bayesian modelling with frequentist assessment. Unlike the PPC, we prove that the HPC is properly calibrated. Empirically, we study HPC on classical regression, a hierarchical model of text data, and factor analysis.<\/jats:p>","DOI":"10.1093\/jrsssb\/qkad105","type":"journal-article","created":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T16:26:56Z","timestamp":1693499216000},"page":"194-214","source":"Crossref","is-referenced-by-count":15,"title":["Holdout predictive checks for Bayesian model criticism"],"prefix":"10.1093","volume":"86","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4438-6350","authenticated-orcid":false,"given":"Gemma E","family":"Moran","sequence":"first","affiliation":[{"name":"Department of Statistics, Rutgers University , Piscataway , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David M","family":"Blei","sequence":"additional","affiliation":[{"name":"Department of Statistics, Department of Computer Science, Columbia Data Science Institute, Columbia University , New York , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajesh","family":"Ranganath","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Center for Data Science, New York University , New York , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,9,15]]},"reference":[{"key":"2024090615151725600_qkad105-B1","first-page":"1127","article-title":"P-values for composite null models","volume":"95","author":"Bayarri","year":"2000","journal-title":"Journal of the American Statistical Association"},{"key":"2024090615151725600_qkad105-B2","first-page":"322","article-title":"Bayesian checking of the second levels of hierarchical models","volume":"22","author":"Bayarri","year":"2007","journal-title":"Statistical Science"},{"key":"2024090615151725600_qkad105-B3","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0378-3758(02)00282-3","article-title":"Bayesian measures of surprise for outlier detection","volume":"111","author":"Bayarri","year":"2003","journal-title":"Journal of Statistical Planning and Inference"},{"key":"2024090615151725600_qkad105-B4","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1080\/01621459.1996.10476668","article-title":"The intrinsic Bayes factor for model selection and prediction","volume":"91","author":"Berger","year":"1996","journal-title":"Journal of the American Statistical Association"},{"key":"2024090615151725600_qkad105-B5","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1146\/annurev-statistics-022513-115657","article-title":"Build, compute, critique, repeat: Data analysis with latent variable models","volume":"1","author":"Blei","year":"2014","journal-title":"Annual Review of Statistics and its Application"},{"key":"2024090615151725600_qkad105-B6","first-page":"993","article-title":"Latent Dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"2024090615151725600_qkad105-B7","doi-asserted-by":"crossref","first-page":"383","DOI":"10.2307\/2982063","article-title":"Sampling and Bayes\u2019 inference in scientific modeling and robustness","volume":"143","author":"Box","year":"1980","journal-title":"Journal of the Royal Statistical Society, Series A"},{"key":"2024090615151725600_qkad105-B8","doi-asserted-by":"crossref","first-page":"816","DOI":"10.1111\/j.1467-9469.2007.00560.x","article-title":"A robust conflict measure of inconsistencies in Bayesian hierarchical models","volume":"34","author":"Dahl","year":"2007","journal-title":"Scandinavian Journal of Statistics"},{"key":"2024090615151725600_qkad105-B9","first-page":"760","article-title":"Comment: Utility,sensitivity analysis, and cross-validation in Bayesian model-checking","volume":"6","author":"Draper","year":"1996","journal-title":"Statistica Sinica"},{"key":"2024090615151725600_qkad105-B10","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1214\/06-BA129","article-title":"Checking for prior-data conflict","volume":"1","author":"Evans","year":"2006","journal-title":"Bayesian Analysis"},{"key":"2024090615151725600_qkad105-B11","doi-asserted-by":"crossref","first-page":"822","DOI":"10.1111\/j.1467-9469.2009.00659.x","article-title":"Extensions of a conflict measure of inconsistencies in Bayesian hierarchical models","volume":"36","author":"G\u00e5semyr","year":"2009","journal-title":"Scandinavian Journal of Statistics"},{"key":"2024090615151725600_qkad105-B12","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","article-title":"The predictive sample reuse method with applications","volume":"70","author":"Geisser","year":"1975","journal-title":"Journal of the American Statistical Association"},{"key":"2024090615151725600_qkad105-B13","article-title":"Model determination using predictive distributions with implementation via sampling-based methods","author":"Gelfand","year":"1992","journal-title":"Technical Report"},{"key":"2024090615151725600_qkad105-B14","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1198\/106186004X11435","article-title":"Exploratory data analysis for complex models","volume":"13","author":"Gelman","year":"2004","journal-title":"Journal of Computational and Graphical Statistics"},{"key":"2024090615151725600_qkad105-B15","doi-asserted-by":"crossref","DOI":"10.1201\/b16018","volume-title":"Bayesian data analysis","author":"Gelman","year":"2013","edition":"3rd ed."},{"key":"2024090615151725600_qkad105-B16","doi-asserted-by":"crossref","DOI":"10.1201\/9780429258411","volume-title":"Bayesian data analysis","author":"Gelman","year":"1995"},{"key":"2024090615151725600_qkad105-B17","first-page":"733","article-title":"Posterior predictive assessment of model fitness via realized discrepancies","volume":"6","author":"Gelman","year":"1996","journal-title":"Statistica Sinica"},{"key":"2024090615151725600_qkad105-B18","author":"Gelman","year":"2020"},{"key":"2024090615151725600_qkad105-B19","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1111\/j.2517-6161.1967.tb00676.x","article-title":"The use of the concept of a future observation in goodness-of-fit problems","volume":"29","author":"Guttman","year":"1967","journal-title":"Journal of the Royal Statistical Society. Series B (Methodological)"},{"key":"2024090615151725600_qkad105-B20","doi-asserted-by":"crossref","first-page":"1157","DOI":"10.1198\/016214505000001393","article-title":"Post-processing posterior predictive p values","volume":"101","author":"Hjort","year":"2006","journal-title":"Journal of the American Statistical Association"},{"key":"2024090615151725600_qkad105-B21","first-page":"1303","article-title":"Stochastic variational inference","volume":"14","author":"Hoffman","year":"2013","journal-title":"Journal of Machine Learning Research"},{"key":"2024090615151725600_qkad105-B22","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1214\/07-BA229","article-title":"Bayesian model assessment using pivotal quantities","volume":"2","author":"Johnson","year":"2007","journal-title":"Bayesian Analysis"},{"key":"2024090615151725600_qkad105-B23","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1214\/07-STS235B","article-title":"Comment: Bayesian checking of the second level of hierarchical models: Cross-validated posterior predictive checks using discrepancy measures","volume":"22","author":"Larsen","year":"2007","journal-title":"Statistical Science"},{"key":"2024090615151725600_qkad105-B24","first-page":"779","article-title":"Comment: Posterior predictive assessment for data subsets in hierarchical models via MCMC","volume":"6","author":"Lewis","year":"1996","journal-title":"Statistica Sinica"},{"key":"2024090615151725600_qkad105-B25","author":"Li","year":"2022"},{"issue":"10","key":"2024090615151725600_qkad105-B26","doi-asserted-by":"crossref","first-page":"1649","DOI":"10.1002\/sim.1403","article-title":"Approximate cross-validatory predictive checks in disease mapping models","volume":"22","author":"Marshall","year":"2003","journal-title":"Statistics in Medicine"},{"key":"2024090615151725600_qkad105-B27","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1214\/07-BA218","article-title":"Identifying outliers in Bayesian hierarchical models: A simulation-based approach","volume":"2","author":"Marshall","year":"2007","journal-title":"Bayesian Analysis"},{"key":"2024090615151725600_qkad105-B28","doi-asserted-by":"crossref","first-page":"1142","DOI":"10.1214\/aos\/1176325622","article-title":"Posterior predictive p-values","volume":"22","author":"Meng","year":"1994","journal-title":"Annals of Statistics"},{"key":"2024090615151725600_qkad105-B29","first-page":"1","article-title":"The posterior predictive null","volume":"1","author":"Moran","year":"2022","journal-title":"Bayesian Analysis"},{"key":"2024090615151725600_qkad105-B30","author":"O\u2019Hagan","year":"2003"},{"key":"2024090615151725600_qkad105-B31","doi-asserted-by":"crossref","first-page":"376","DOI":"10.1214\/13-STS426","article-title":"Conflict diagnostics in directed acyclic graphs, with applications in Bayesian evidence synthesis","volume":"28","author":"Presanis","year":"2013","journal-title":"Statistical Science"},{"key":"2024090615151725600_qkad105-B32","first-page":"1143","article-title":"Asymptotic distribution of p-values in composite null models","volume":"95","author":"Robins","year":"2000","journal-title":"Journal of the American Statistical Association"},{"key":"2024090615151725600_qkad105-B33","doi-asserted-by":"crossref","first-page":"1151","DOI":"10.1214\/aos\/1176346785","article-title":"Bayesianly justifiable and relevant frequency calculations for the applied statistician","volume":"12","author":"Rubin","year":"1984","journal-title":"Annals of Statistics"},{"key":"2024090615151725600_qkad105-B34","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1111\/1467-9868.00353","article-title":"Bayesian measures of model complexity and fit","volume":"64","author":"Spiegelhalter","year":"2002","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"key":"2024090615151725600_qkad105-B35","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1111\/1467-9868.00196","article-title":"Probabilistic principal component analysis","volume":"61","author":"Tipping","year":"1999","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2024090615151725600_qkad105-B36","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511800474","volume-title":"Algebraic geometry and statistical learning theory (Vol. 25)","author":"Watanabe","year":"2009"},{"key":"2024090615151725600_qkad105-B37","first-page":"3571","article-title":"Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory","volume":"11","author":"Watanabe","year":"2010","journal-title":"Journal of Machine Learning Research"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/1\/194\/59034474\/qkad105.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/1\/194\/59034474\/qkad105.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,6]],"date-time":"2024-09-06T21:11:35Z","timestamp":1725657095000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/86\/1\/194\/7274627"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,15]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,9,15]]},"published-print":{"date-parts":[[2024,2,8]]}},"URL":"https:\/\/doi.org\/10.1093\/jrsssb\/qkad105","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,2]]},"published":{"date-parts":[[2023,9,15]]}}}