{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T20:06:20Z","timestamp":1774123580808,"version":"3.50.1"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T00:00:00Z","timestamp":1716249600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T00:00:00Z","timestamp":1716249600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001779","name":"Monash University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001779","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Stat Comput"],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Brute force cross-validation (CV) is a method for predictive assessment and model selection that is general and applicable to a wide range of Bayesian models. Naive or \u2018brute force\u2019 CV approaches are often too computationally costly for interactive modeling workflows, especially when inference relies on Markov chain Monte Carlo (MCMC). We propose overcoming this limitation using massively parallel MCMC. Using accelerator hardware such as graphics processor units, our approach can be about as fast (in wall clock time) as a single full-data model fit. Parallel CV is flexible because it can easily exploit a wide range data partitioning schemes, such as those designed for non-exchangeable data. It can also accommodate a range of scoring rules. We propose MCMC diagnostics, including a summary of MCMC mixing based on the popular potential scale reduction factor (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\widehat{\\textrm{R}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mover>\n                    <mml:mtext>R<\/mml:mtext>\n                    <mml:mo>^<\/mml:mo>\n                  <\/mml:mover>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) and MCMC effective sample size (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\widehat{\\textrm{ESS}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mover>\n                    <mml:mtext>ESS<\/mml:mtext>\n                    <mml:mo>^<\/mml:mo>\n                  <\/mml:mover>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) measures. We also describe a method for determining whether an <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\widehat{\\textrm{R}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mover>\n                    <mml:mtext>R<\/mml:mtext>\n                    <mml:mo>^<\/mml:mo>\n                  <\/mml:mover>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> diagnostic indicates approximate stationarity of the chains, that may be of more general interest for applications beyond parallel CV. Finally, we show that parallel CV and its diagnostics can be implemented with online algorithms, allowing parallel CV to scale up to very large blocking designs on memory-constrained computing accelerators.<\/jats:p>","DOI":"10.1007\/s11222-024-10404-w","type":"journal-article","created":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T03:32:06Z","timestamp":1716262326000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Bayesian cross-validation by parallel Markov chain Monte Carlo"],"prefix":"10.1007","volume":"34","author":[{"given":"Alex","family":"Cooper","sequence":"first","affiliation":[]},{"given":"Aki","family":"Vehtari","sequence":"additional","affiliation":[]},{"given":"Catherine","family":"Forbes","sequence":"additional","affiliation":[]},{"given":"Dan","family":"Simpson","sequence":"additional","affiliation":[]},{"given":"Lauren","family":"Kennedy","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,5,21]]},"reference":[{"key":"10404_CR1","unstructured":"Chen, T., Fox, E., Guestrin, C.: Stochastic gradient Hamiltonian Monte Carlo. In: International conference on machine learning, PMLR, pp 1683\u20131691 (2014)"},{"key":"10404_CR2","doi-asserted-by":"publisher","unstructured":"Cooper, A., Simpson, D., Kennedy, L., et al.: Cross-validatory model selection for Bayesian autoregressions with exogenous regressors Bayesian Anal. Advance Publication (2024). https:\/\/doi.org\/10.1214\/23-BA1409","DOI":"10.1214\/23-BA1409"},{"issue":"1","key":"10404_CR3","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1080\/00949659908811968","volume":"64","author":"MK Cowles","year":"1999","unstructured":"Cowles, M.K., Roberts, G.O., Rosenthal, J.S.: Possible biases induced by MCMC convergence diagnostics. J. Stat. Comput. Simul. 64(1), 87\u2013104 (1999). https:\/\/doi.org\/10.1080\/00949659908811968","journal-title":"J. Stat. Comput. Simul."},{"issue":"2","key":"10404_CR4","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1007\/s40300-014-0039-y","volume":"72","author":"AP Dawid","year":"2014","unstructured":"Dawid, A.P., Musio, M.: Theory and applications of proper scoring rules. METRON 72(2), 169\u2013183 (2014). https:\/\/doi.org\/10.1007\/s40300-014-0039-y","journal-title":"METRON"},{"issue":"2","key":"10404_CR5","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1214\/15-BA942","volume":"10","author":"AP Dawid","year":"2015","unstructured":"Dawid, A.P., Musio, M.: Bayesian model selection based on proper scoring rules. Bayesian Anal. 10(2), 479\u2013499 (2015). https:\/\/doi.org\/10.1214\/15-BA942","journal-title":"Bayesian Anal."},{"issue":"1","key":"10404_CR6","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1214\/aos\/1018031101","volume":"27","author":"AP Dawid","year":"1999","unstructured":"Dawid, A.P., Sebastiani, P.: Coherent dispersion criteria for optimal experimental design. Ann. Stat. 27(1), 65\u201381 (1999)","journal-title":"Ann. Stat."},{"key":"10404_CR7","unstructured":"Dillon, J.V., Langmore, I., Tran, D., et\u00a0al.: TensorFlow distributions, (2017). arXiv:1711.10604"},{"issue":"248","key":"10404_CR8","first-page":"1","volume":"24","author":"P Gagnon","year":"2023","unstructured":"Gagnon, P., Maire, F., Zanella, G.: Improving multiple-try metropolis with local balancing. J. Mach. Learn. Res. 24(248), 1\u201359 (2023)","journal-title":"J. Mach. Learn. Res."},{"issue":"350","key":"10404_CR9","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","volume":"70","author":"S Geisser","year":"1975","unstructured":"Geisser, S.: The predictive sample reuse method with applications. J. Am. Stat. Assoc. 70(350), 320\u2013328 (1975). https:\/\/doi.org\/10.1080\/01621459.1975.10479865","journal-title":"J. Am. Stat. Assoc."},{"issue":"412","key":"10404_CR10","doi-asserted-by":"publisher","first-page":"972","DOI":"10.1080\/01621459.1990.10474968","volume":"85","author":"AE Gelfand","year":"1990","unstructured":"Gelfand, A.E., Hills, S.E., Racine-Poon, A., et al.: Illustration of Bayesian inference in normal data models using Gibbs sampling. J. Am. Stat. Assoc. 85(412), 972\u2013985 (1990)","journal-title":"J. Am. Stat. Assoc."},{"key":"10404_CR11","first-page":"147","volume":"4","author":"AE Gelfand","year":"1992","unstructured":"Gelfand, A.E., Dey, D.K., Chang, H.: Model determination using predictive distributions, with implementation via sampling-based methods (with discussion). Bayesian Stat. 4, 147 (1992)","journal-title":"Bayesian Stat."},{"key":"10404_CR12","doi-asserted-by":"crossref","unstructured":"Gelman, A., Hill, J.: Data Analysis Using Regression and Multilevel\/Hierarchical Models. Cambridge University Press (2006)","DOI":"10.1017\/CBO9780511790942"},{"issue":"4","key":"10404_CR13","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1214\/ss\/1177011136","volume":"7","author":"A Gelman","year":"1992","unstructured":"Gelman, A., Rubin, D.B.: Inference from iterative simulation using multiple sequences. Stat. Sci. 7(4), 457\u2013472 (1992)","journal-title":"Stat. Sci."},{"key":"10404_CR14","volume-title":"Bayesian Data Analysis","author":"A Gelman","year":"2014","unstructured":"Gelman, A., Carlin, J.B., Stern, H.S., et al.: Bayesian Data Analysis, 3rd edn. Chapman & Hall\/CRC, Boca Raton (2014)","edition":"3"},{"key":"10404_CR15","unstructured":"Gelman, A., Vehtari, A., Simpson, D., et\u00a0al.: Bayesian workflow (2020). Preprint at arXiv:2011.01808,"},{"key":"10404_CR16","doi-asserted-by":"crossref","unstructured":"Geyer, C.J.: Practical Markov chain Monte Carlo. Stat. Sci. pp 473\u2013483 (1992)","DOI":"10.1214\/ss\/1177011137"},{"key":"10404_CR17","unstructured":"Glatt-Holtz, N.E., Holbrook, A.J., Krometis, J.A., et\u00a0al.: Parallel MCMC algorithms: Theoretical foundations, algorithm design, case studies (2022). arXiv preprint arXiv:2209.04750"},{"issue":"477","key":"10404_CR18","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102(477), 359\u2013378 (2007). https:\/\/doi.org\/10.1198\/016214506000001437","journal-title":"J. Am. Stat. Assoc."},{"issue":"1","key":"10404_CR19","first-page":"1593","volume":"15","author":"MD Hoffman","year":"2014","unstructured":"Hoffman, M.D., Gelman, A., et al.: The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res. 15(1), 1593\u20131623 (2014)","journal-title":"J. Mach. Learn. Res."},{"key":"10404_CR20","doi-asserted-by":"publisher","first-page":"1402","DOI":"10.1080\/10618600.2023.2195890","volume":"32","author":"AJ Holbrook","year":"2023","unstructured":"Holbrook, A.J.: A quantum parallel Markov chain Monte Carlo. J. Comput. Graph. Stat. 32, 1402 (2023)","journal-title":"J. Comput. Graph. Stat."},{"issue":"4","key":"10404_CR21","first-page":"695","volume":"6","author":"A Hyv\u00e4rinen","year":"2005","unstructured":"Hyv\u00e4rinen, A., Dayan, P.: Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res. 6(4), 695 (2005)","journal-title":"J. Mach. Learn. Res."},{"issue":"476","key":"10404_CR22","doi-asserted-by":"publisher","first-page":"1537","DOI":"10.1198\/016214506000000492","volume":"101","author":"GL Jones","year":"2006","unstructured":"Jones, G.L., Haran, M., Caffo, B.S., et al.: Fixed-width output analysis for Markov chain Monte Carlo. J. Am. Stat. Assoc. 101(476), 1537\u20131547 (2006). https:\/\/doi.org\/10.1198\/016214506000000492","journal-title":"J. Am. Stat. Assoc."},{"issue":"2","key":"10404_CR23","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1111\/insr.12405","volume":"89","author":"F Kr\u00fcger","year":"2021","unstructured":"Kr\u00fcger, F., Lerch, S., Thorarinsdottir, T., et al.: Predictive inference based on Markov chain Monte Carlo output. Int. Stat. Rev. 89(2), 274\u2013301 (2021). https:\/\/doi.org\/10.1111\/insr.12405","journal-title":"Int. Stat. Rev."},{"key":"10404_CR24","doi-asserted-by":"crossref","unstructured":"Kumar, R., Carroll, C., Hartikainen, A., et al.: ArviZ: a unified library for exploratory analysis of Bayesian models in python. J. Open Source Softw. (2019)","DOI":"10.21105\/joss.01143"},{"key":"10404_CR25","unstructured":"Lao, J., Louf, R.: Blackjax: A sampling library for JAX, (2020). http:\/\/github.com\/blackjax-devs\/blackjax"},{"key":"10404_CR26","unstructured":"Lao, J., Suter, C., Langmore, I., et\u00a0al.: tfp.mcmc: Modern Markov Chain Monte Carlo Tools Built for Modern Hardware (2020). Preprint at arXiv:2002.01184"},{"key":"10404_CR27","unstructured":"Magnusson, M., Vehtari, A., Jonasson, J., et\u00a0al.: Leave-one-out cross-validation for Bayesian model comparison in large data. In: Chiappa S, Calandra R (eds) Proceedings of the twenty third international conference on artificial intelligence and statistics, proceedings of machine learning research, vol 108. PMLR, pp 341\u2013351 (2020), https:\/\/proceedings.mlr.press\/v108\/magnusson20a.html"},{"key":"10404_CR28","unstructured":"Mahoney, M.J., Johnson, L.K., Silge, J., et\u00a0al.: Assessing the performance of spatial cross-validation approaches for models of spatially structured data (2023). Preprint at arXiv:2303.07334"},{"key":"10404_CR29","unstructured":"Margossian, C.C., Hoffman, M.D., Sountsov, P., et al.: Nested$${\\hat{R}}$$: Assessing the convergence of Markov chain Monte Carlo when running many short chains (2023). Preprint at arXiv:2110.13017"},{"key":"10404_CR30","doi-asserted-by":"publisher","unstructured":"Moins, T., Arbel, J., Dutfoy, A., et al.: On the use of a local $${{\\widehat{r}}}$$ to improve mcmc convergence diagnostic. Bayesian Anal. Advance Publication. (2023) https:\/\/doi.org\/10.1214\/23-BA1399","DOI":"10.1214\/23-BA1399"},{"key":"10404_CR31","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1201\/b10905-6","volume-title":"Handbook of Markov Chain Monte Carlo","author":"RM Neal","year":"2011","unstructured":"Neal, R.M.: MCMC using Hamiltonian dynamics. In: Handbook of Markov Chain Monte Carlo, vol. 2, pp. 113\u2013162. CRC Press, New York (2011)"},{"key":"10404_CR32","unstructured":"Neiswanger, W., Wang, C., Xing, E.P.: Asymptotically exact, embarrassingly parallel MCMC. In: Proceedings of the Thirtieth conference on uncertainty in artificial intelligence. AUAI Press, Arlington, Virginia, USA, UAI\u201914, p 623\u2013632 (2014)"},{"key":"10404_CR33","unstructured":"Nissen, J.N.: What scientists must know about hardware to write fast code. https:\/\/viralinstruction.com\/posts\/hardware\/, Accessed 09 Jan 2023 (2023)"},{"issue":"1","key":"10404_CR34","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1016\/S0304-4076(00)00030-0","volume":"99","author":"J Racine","year":"2000","unstructured":"Racine, J.: Consistent cross-validatory model-selection for dependent data: hv-block cross-validation. J. Econ. 99(1), 39\u201361 (2000). https:\/\/doi.org\/10.1016\/S0304-4076(00)00030-0","journal-title":"J. Econ."},{"key":"10404_CR35","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-4145-2","volume-title":"Monte Carlo Statistical Methods","author":"CP Robert","year":"2004","unstructured":"Robert, C.P., Casella, G.: Monte Carlo Statistical Methods. Springer, New York (2004)"},{"issue":"8","key":"10404_CR36","doi-asserted-by":"publisher","first-page":"913","DOI":"10.1111\/ecog.02881","volume":"40","author":"DR Roberts","year":"2017","unstructured":"Roberts, D.R., Bahn, V., Ciuti, S., et al.: Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography 40(8), 913\u2013929 (2017)","journal-title":"Ecography"},{"key":"10404_CR37","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1146\/annurev-statistics-031219-041300","volume":"7","author":"V Roy","year":"2020","unstructured":"Roy, V.: Convergence diagnostics for Markov chain Monte Carlo. Ann. Rev. Stat. Appl. 7, 387\u2013412 (2020)","journal-title":"Ann. Rev. Stat. Appl."},{"key":"10404_CR38","doi-asserted-by":"publisher","unstructured":"Rubin, D.B.: The Bayesian bootstrap. Ann. Stat. 9(1), 130\u2013134 (1981). https:\/\/doi.org\/10.1214\/aos\/1176345338","DOI":"10.1214\/aos\/1176345338"},{"issue":"2","key":"10404_CR39","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1080\/17509653.2016.1142191","volume":"11","author":"S Scott","year":"2016","unstructured":"Scott, S., Blocker, A., Bonassi, F., et al.: Bayes and big data: the consensus Monte Carlo algorithm. Int. J. Manag. Sci. Eng. Manag. 11(2), 78\u201388 (2016). https:\/\/doi.org\/10.1080\/17509653.2016.1142191","journal-title":"Int. J. Manag. Sci. Eng. Manag."},{"key":"10404_CR40","unstructured":"Sivula, T., Magnusson, M., Matamoros, A.A., et\u00a0al.: Uncertainty in Bayesian leave-one-out cross-validation based model comparison, (2022). Preprint at arXiv:2008.10296"},{"key":"10404_CR41","volume-title":"Computer Organization and Architecture","author":"W Stallings","year":"2015","unstructured":"Stallings, W.: Computer Organization and Architecture. Pearson Education, Limited (2015)"},{"key":"10404_CR42","unstructured":"Stan Development Team: Stan Modeling Language: User\u2019s Guide and Reference Manual (2022)"},{"issue":"10","key":"10404_CR43","doi-asserted-by":"publisher","first-page":"2439","DOI":"10.1162\/08997660260293292","volume":"14","author":"A Vehtari","year":"2002","unstructured":"Vehtari, A., Lampinen, J.: Bayesian model assessment and comparison using cross-validation predictive densities. Neural Comput. 14(10), 2439\u20132468 (2002). https:\/\/doi.org\/10.1162\/08997660260293292","journal-title":"Neural Comput."},{"issue":"5","key":"10404_CR44","doi-asserted-by":"publisher","first-page":"1413","DOI":"10.1007\/s11222-016-9696-4","volume":"27","author":"A Vehtari","year":"2017","unstructured":"Vehtari, A., Gelman, A., Gabry, J.: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statist. Comput. 27(5), 1413\u20131432 (2017). https:\/\/doi.org\/10.1007\/s11222-016-9696-4","journal-title":"Statist. Comput."},{"issue":"1","key":"10404_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/20-ba1221","volume":"1","author":"A Vehtari","year":"2020","unstructured":"Vehtari, A., Gelman, A., Simpson, D., et al.: Rank-normalization, folding, and localization: an improved $${\\widehat{R}}$$ for assessing convergence of MCMC. Bayesian Anal. 1(1), 1\u201326 (2020). https:\/\/doi.org\/10.1214\/20-ba1221","journal-title":"Bayesian Anal."},{"key":"10404_CR46","unstructured":"Vehtari, A., Gelman, A., Sivula, T., et al.: Expectation propagation as a way of life: a framework for Bayesian inference on partitioned data. J. Mach. Learn. Res. 21(17), 1\u201353 (2020)"}],"container-title":["Statistics and Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11222-024-10404-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11222-024-10404-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11222-024-10404-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,13]],"date-time":"2024-07-13T07:11:42Z","timestamp":1720854702000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11222-024-10404-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,21]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10404"],"URL":"https:\/\/doi.org\/10.1007\/s11222-024-10404-w","relation":{},"ISSN":["0960-3174","1573-1375"],"issn-type":[{"value":"0960-3174","type":"print"},{"value":"1573-1375","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,21]]},"assertion":[{"value":"23 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 February 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"119"}}