{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T00:00:42Z","timestamp":1773446442500,"version":"3.50.1"},"reference-count":124,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T00:00:00Z","timestamp":1720137600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T00:00:00Z","timestamp":1720137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"crossref","award":["200020_214879"],"award-info":[{"award-number":["200020_214879"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Statistical learning algorithms provide a generally-applicable framework to sidestep time-consuming experiments, or accurate physics-based modeling, but they introduce a further source of error on top of the intrinsic limitations of the experimental or theoretical setup. Uncertainty estimation is essential to quantify this error, and to make application of data-centric approaches more trustworthy. To ensure that uncertainty quantification is used widely, one should aim for algorithms that are accurate, but also easy to implement and apply. In particular, including uncertainty quantification on top of an existing architecture should be straightforward, and add minimal computational overhead. Furthermore, it should be easy to manipulate or combine multiple machine-learning predictions, propagating uncertainty over further modeling steps. We compare several well-established uncertainty quantification frameworks against these requirements, and propose a practical approach, which we dub direct propagation of shallow ensembles, that provides a good compromise between ease of use and accuracy. We present benchmarks for generic datasets, and an in-depth study of applications to the field of atomistic machine learning for chemistry and materials. These examples underscore the importance of using a formulation that allows propagating errors without making strong assumptions on the correlations between different predictions of the model.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad594a","type":"journal-article","created":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T19:02:00Z","timestamp":1718650920000},"page":"035006","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Uncertainty quantification by direct propagation of shallow ensembles"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-3849-7813","authenticated-orcid":true,"given":"Matthias","family":"Kellner","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2571-2832","authenticated-orcid":true,"given":"Michele","family":"Ceriotti","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2024,7,5]]},"reference":[{"key":"mlstad594abib1","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.91.045002","article-title":"Machine learning and the physical sciences","volume":"91","author":"Carleo","year":"2019","journal-title":"Rev. Mod. Phys."},{"key":"mlstad594abib2","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1038\/s41592-019-0666-6","article-title":"Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning","volume":"17","author":"Gainza","year":"2020","journal-title":"Nat. Methods"},{"key":"mlstad594abib3","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1038\/s41586-021-04301-9","article-title":"Magnetic control of tokamak plasmas through deep reinforcement learning","volume":"602","author":"Degrave","year":"2022","journal-title":"Nature"},{"key":"mlstad594abib4","doi-asserted-by":"publisher","DOI":"10.1063\/1.4966192","article-title":"Perspective: machine learning potentials for atomistic simulations","volume":"145","author":"Behler","year":"2016","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib5","doi-asserted-by":"publisher","first-page":"1045","DOI":"10.1557\/s43577-022-00440-0","article-title":"Beyond potentials: integrated machine learning models for materials","volume":"47","author":"Ceriotti","year":"2022","journal-title":"MRS Bull."},{"key":"mlstad594abib6","doi-asserted-by":"crossref","DOI":"10.1109\/SC41405.2020.00009","article-title":"Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning","author":"Jia","year":"2020"},{"key":"mlstad594abib7","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1038\/s41586-020-03072-z","article-title":"Origins of structural and electronic transitions in disordered silicon","volume":"589","author":"Deringer","year":"2021","journal-title":"Nature"},{"key":"mlstad594abib8","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1038\/s41928-023-01030-x","article-title":"Device-scale atomistic modelling of phase-change memory materials","volume":"6","author":"Zhou","year":"2023","journal-title":"Nat. Electron."},{"key":"mlstad594abib9","article-title":"Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic size","author":"Musaelian","year":"2023"},{"key":"mlstad594abib10","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.98.146401","article-title":"Generalized neural-network representation of high-dimensional potential-energy surfaces","volume":"98","author":"Behler","year":"2007","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib11","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.104.136403","article-title":"Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons","volume":"104","author":"Bart\u00f3k","year":"2010","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib12","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.108.058301","article-title":"Fast and accurate modeling of molecular atomization energies with machine learning","volume":"108","author":"Rupp","year":"2012","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib13","doi-asserted-by":"publisher","first-page":"9719","DOI":"10.1021\/acs.chemrev.1c00598","article-title":"Introduction: machine learning at the atomic scale","volume":"121","author":"Ceriotti","year":"2021","journal-title":"Chem. Rev."},{"key":"mlstad594abib14","first-page":"pp 287","article-title":"Query by committee","author":"Seung","year":"1992"},{"key":"mlstad594abib15","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.100.014105","article-title":"On-the-fly machine learning force field generation: application to melting points","volume":"100","author":"Jinnouchi","year":"2019","journal-title":"Phys. Rev. B"},{"key":"mlstad594abib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41524-020-0283-z","article-title":"On-the-fly active learning of interpretable Bayesian force fields for atomistic rare events","volume":"6","author":"Vandermause","year":"2020","journal-title":"npj Comput. Mater."},{"key":"mlstad594abib17","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/j.inffus.2021.05.008","article-title":"A review of uncertainty quantification in deep learning: techniques, applications and challenges","volume":"76","author":"Abdar","year":"2021","journal-title":"Inf. Fusion"},{"key":"mlstad594abib18","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","article-title":"Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods","volume":"110","author":"H\u00fcllermeier","year":"2021","journal-title":"Mach. Learn."},{"key":"mlstad594abib19","doi-asserted-by":"publisher","DOI":"10.1063\/5.0067565","article-title":"Learning intermolecular forces at liquid-vapor interfaces","volume":"155","author":"Niblett","year":"2021","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib20","doi-asserted-by":"publisher","DOI":"10.1063\/5.0142843","article-title":"A \u201cshort blanket\u201d dilemma for a state-of-the-art neural network potential for water: reproducing experimental properties or the physics of the underlying many-body interactions?","volume":"158","author":"Zhai","year":"2023","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib21","article-title":"Physics-inspired equivariant descriptors of non-bonded interactions","author":"Huguenin-Dumittan","year":"2023"},{"key":"mlstad594abib22","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.125.166001","article-title":"Incompleteness of atomic structure representations","volume":"125","author":"Pozdnyakov","year":"2020","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib23","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/aca1f8","article-title":"Incompleteness of graph neural networks for points clouds in three dimensions","volume":"3","author":"Pozdnyakov","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib24","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac3eb3","article-title":"Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks","volume":"3","author":"Busk","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib25","doi-asserted-by":"publisher","DOI":"10.1039\/D3CP02143B","article-title":"Graph neural network interatomic potential ensembles with calibrated aleatoric and epistemic uncertainty on energy and forces","volume":"25","author":"Busk","year":"2023","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstad594abib26","doi-asserted-by":"publisher","first-page":"4520","DOI":"10.1021\/acs.jctc.2c01267","article-title":"Scalable Bayesian uncertainty quantification for neural network potentials: promise and pitfalls","volume":"19","author":"Thaler","year":"2023","journal-title":"J. Chem. Theory Comput."},{"key":"mlstad594abib27","doi-asserted-by":"crossref","DOI":"10.1038\/s41524-023-01180-8","article-title":"Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles","author":"Tan","year":"2023"},{"key":"mlstad594abib28","doi-asserted-by":"publisher","first-page":"7913","DOI":"10.1039\/C9SC02298H","article-title":"A quantitative uncertainty metric controls error in neural network-driven chemical discovery","volume":"10","author":"Paul Janet","year":"2019","journal-title":"Chem. Sci."},{"key":"mlstad594abib29","doi-asserted-by":"publisher","DOI":"10.1063\/5.0136574","article-title":"Fast uncertainty estimates in deep learning interatomic potentials","volume":"158","author":"Zhu","year":"2023","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib30","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/aca7b1","article-title":"Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials","volume":"3","author":"Hu","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib31","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1021\/acs.jctc.8b00959","article-title":"Fast and accurate uncertainty estimation in chemical machine learning","volume":"15","author":"Musil","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstad594abib32","doi-asserted-by":"publisher","DOI":"10.1063\/5.0146905","article-title":"Deep ensembles vs committees for uncertainty estimation in neural-network force fields: comparison and application to active learning","volume":"158","author":"Carrete","year":"2023","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib33","doi-asserted-by":"publisher","DOI":"10.1039\/D2SC04056E","article-title":"Uncertainty quantification for predictions of atomistic neural networks","volume":"13","author":"Itza Vazquez-Salazar","year":"2022","journal-title":"Chem. Sci."},{"key":"mlstad594abib34","doi-asserted-by":"publisher","DOI":"10.1088\/1361-651X\/ab0d75","article-title":"Uncertainty quantification for classical effective potentials: an extension to potfit","volume":"27","author":"Longbottom","year":"2019","journal-title":"Modelling Simul. Mater. Sci. Eng."},{"key":"mlstad594abib35","article-title":"High accuracy uncertainty-aware interatomic force modeling with equivariant Bayesian neural networks","author":"Rensmeyer","year":"2023"},{"key":"mlstad594abib36","doi-asserted-by":"publisher","first-page":"5129","DOI":"10.1021\/acs.jpca.0c02395","article-title":"Bayesian machine learning approach to the quantification of uncertainties on Ab Initio potential energy surfaces","volume":"124","author":"Venturi","year":"2020","journal-title":"J. Phys. Chem. A"},{"key":"mlstad594abib37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41524-020-00390-8","article-title":"Uncertainty quantification in molecular simulations with dropout neural network potentials","volume":"6","author":"Wen","year":"2020","journal-title":"npj Comput. Mater."},{"key":"mlstad594abib38","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/abe294","article-title":"Exploration of transferable and uniformly accurate neural network interatomic potentials using optimal experimental design","volume":"2","author":"Zaverkin","year":"2021","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib39","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.105.015311","article-title":"Quality of uncertainty estimates from neural network potential ensembles","volume":"105","author":"Kahle","year":"2022","journal-title":"Phys. Rev. E"},{"key":"mlstad594abib40","doi-asserted-by":"publisher","DOI":"10.1039\/C7CP00375G","article-title":"Addressing uncertainty in atomistic machine learning","volume":"19","author":"Peterson","year":"2017","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstad594abib41","doi-asserted-by":"publisher","DOI":"10.1063\/5.0109572","article-title":"Prediction uncertainty validation for computational chemists","volume":"157","author":"Pernot","year":"2022","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib42","doi-asserted-by":"publisher","DOI":"10.1063\/5.0174943","article-title":"Calibration in machine learning uncertainty quantification: beyond consistency to target adaptivity","volume":"1","author":"Pernot","year":"2023","journal-title":"APL Mach. Learn."},{"key":"mlstad594abib43","article-title":"Improved uncertainty quantification for Gaussian process regression based interatomic potentials","author":"Bart\u00f3k","year":"2022"},{"key":"mlstad594abib44","article-title":"Statistical methods for resolving poor uncertainty quantification in machine learning interatomic potentials","author":"Annevelink","year":"2023"},{"key":"mlstad594abib45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41524-023-00988-8","article-title":"Uncertainty-aware molecular dynamics from Bayesian active learning for phase transformations and thermal transport in SiC","volume":"9","author":"Xie","year":"2023","journal-title":"npj Comput. Mater."},{"key":"mlstad594abib46","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1186\/s13321-023-00682-3","article-title":"Explainable uncertainty quantifications for deep learning-based molecular property prediction","volume":"15","author":"Yang","year":"2023","journal-title":"J. Cheminf."},{"key":"mlstad594abib47","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ad0ab5","article-title":"Calibration of uncertainty in the active learning of machine learning force fields","volume":"4","author":"Thomas-Mitchell","year":"2023","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib48","doi-asserted-by":"publisher","first-page":"3846","DOI":"10.1021\/acs.jcim.1c00670","article-title":"Pairwise difference regression: a machine learning meta-algorithm for improved prediction and uncertainty quantification in chemical search","volume":"61","author":"Tynes","year":"2021","journal-title":"J. Chem. Inf. Model."},{"key":"mlstad594abib49","doi-asserted-by":"publisher","first-page":"2697","DOI":"10.1021\/acs.jcim.9b00975","article-title":"Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction","volume":"60","author":"Scalia","year":"2020","journal-title":"J. Chem. Inf. Model."},{"key":"mlstad594abib50","doi-asserted-by":"publisher","DOI":"10.1088\/1361-648X\/ac1280","article-title":"Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet","volume":"33","author":"Breuck","year":"2021","journal-title":"J. Phys.: Condens. Matter"},{"key":"mlstad594abib51","first-page":"p 177","author":"Rasmussen","year":"2008","edition":"3rd edn"},{"key":"mlstad594abib52","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1162\/neco.1996.8.4.843","article-title":"Using neural networks to model conditional multivariate densities","volume":"8","author":"Williams","year":"1996","journal-title":"Neural Comput."},{"key":"mlstad594abib53","first-page":"pp 14927","article-title":"Deep evidential regression","volume":"vol 33","author":"Amini","year":"2020"},{"key":"mlstad594abib54","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1111\/j.2153-3490.1969.tb00483.x","article-title":"Stochastic dynamic prediction","volume":"21","author":"Epstein","year":"1969","journal-title":"Tellus"},{"key":"mlstad594abib55","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1175\/1520-0450(1969)0082.0.CO;2","article-title":"The role of initial uncertainties in predicion","volume":"8","author":"Epstein","year":"1969","journal-title":"J. Appl. Meteorol. Climatol."},{"key":"mlstad594abib56","doi-asserted-by":"publisher","first-page":"1155","DOI":"10.1175\/MWR2906.1","article-title":"Using Bayesian model averaging to calibrate forecast ensembles","volume":"133","author":"Raftery","year":"2005","journal-title":"Mon. Weather Rev."},{"key":"mlstad594abib57","doi-asserted-by":"publisher","first-page":"2317","DOI":"10.1175\/1520-0477(1993)0742.0.CO;2","article-title":"Ensemble forecasting at NMC: the generation of perturbations","volume":"74","author":"Toth","year":"1993","journal-title":"Bull. Am. Meteorol. Soc."},{"key":"mlstad594abib58","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.1175\/MWR2904.1","article-title":"Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation","volume":"133","author":"Gneiting","year":"2005","journal-title":"Mon. Weather Rev."},{"key":"mlstad594abib59","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"vol 30","author":"Lakshminarayanan","year":"2017"},{"key":"mlstad594abib60","first-page":"pp 55","article-title":"Estimating the mean and variance of the target probability distribution","volume":"vol 1","author":"Nix","year":"1994"},{"key":"mlstad594abib61","article-title":"Why M heads are better than one: training adiverse ensemble of deep networks","author":"Lee","year":"2015"},{"key":"mlstad594abib62","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules,prediction and estimation","volume":"102","author":"Gneiting","year":"2007","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstad594abib63","article-title":"Uncertainty toolbox: an open-source library for assessing,visualizing, and improving uncertainty quantification","author":"Chung","year":"2021"},{"key":"mlstad594abib64","doi-asserted-by":"publisher","DOI":"10.1063\/5.0084302","article-title":"The long road to calibrated prediction uncertainty in computational chemistry","volume":"156","author":"Pernot","year":"2022","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib65","article-title":"Evaluating and calibrating uncertainty prediction in regression tasks","author":"Levi","year":"2020"},{"key":"mlstad594abib66","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ab7e1a","article-title":"Methods for comparing uncertainty quantifications for material property predictions","volume":"1","author":"Tran","year":"2020","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib67","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","article-title":"Probabilistic forecasts, calibration and sharpness","volume":"69","author":"Gneiting","year":"2007","journal-title":"J. R. Stat. Soc. B"},{"key":"mlstad594abib68","first-page":"pp 1321","article-title":"On calibration of modern neural networks","author":"Guo","year":"2017"},{"key":"mlstad594abib69","first-page":"pp 2796","article-title":"Accurate uncertainties for deep learning using calibrated regression","author":"Kuleshov","year":"2018"},{"key":"mlstad594abib70","article-title":"On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks","author":"Seitzer","year":"2021"},{"key":"mlstad594abib71","first-page":"pp 7089","article-title":"On double-descent in uncertainty quantification in overparametrized models","author":"Clarte","year":"2023"},{"key":"mlstad594abib72","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/acd749","article-title":"Theoretical characterization of uncertainty in high-dimensional linear classification","volume":"4","author":"Clart\u00e9","year":"2023","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad594abib73","doi-asserted-by":"publisher","DOI":"10.1063\/5.0036522","article-title":"Uncertainty estimation for molecular dynamics and sampling","volume":"154","author":"Imbalzano","year":"2021","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib74","article-title":"Parametric and multivariate uncertainty calibration for regression and object detection","author":"K\u00fcppers","year":"2022"},{"key":"mlstad594abib75","doi-asserted-by":"publisher","first-page":"e1539","DOI":"10.1002\/wics.1539","article-title":"Modern Monte Carlo methods for efficient uncertainty quantification and propagation: a survey","volume":"13","author":"Zhang","year":"2021","journal-title":"WIREs Comput. Stat."},{"key":"mlstad594abib76","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1198\/0003130032413","article-title":"A history of the metropolis-hastings algorithm","volume":"57","author":"Hitchcock","year":"2003","journal-title":"Am. Stat."},{"key":"mlstad594abib77","doi-asserted-by":"publisher","first-page":"627","DOI":"10.1093\/ije\/dyt043","article-title":"Markov Chain Monte Carlo: an introduction for epidemiologists","volume":"42","author":"Hamra","year":"2013","journal-title":"Int. J. Epidemiol."},{"key":"mlstad594abib78","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1016\/0021-9991(77)90121-8","article-title":"Nonphysical sampling distributions in Monte Carlo free-energy estimation: umbrella sampling","volume":"23","author":"Torrie","year":"1977","journal-title":"J. Comput. Phys."},{"key":"mlstad594abib79","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1098\/rspa.2011.0413","article-title":"The inefficiency of re-weighted sampling and the curse of system size in high-order path integration","volume":"468","author":"Ceriotti","year":"2012","journal-title":"Proc. R. Soc. Math. Phys. Eng. Sci."},{"key":"mlstad594abib80","article-title":"The original collection of datasets, available from [122\u2013124], also include results for the Boston housing dataset, which has since been deprecated because of the presence of racial biases. We include them for consistency with the original publication, as we use them for applications with no ethical or policy implications."},{"key":"mlstad594abib81","first-page":"pp 1861","article-title":"Probabilistic backpropagation for scalable learning of Bayesian neural networks","author":"Hernandez-Lobato","year":"2015"},{"key":"mlstad594abib82","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.87.184115","article-title":"On representing chemical environments","volume":"87","author":"Bart\u00f3k","year":"2013","journal-title":"Phys. Rev. B"},{"key":"mlstad594abib83","author":"Fraux","year":"2024"},{"key":"mlstad594abib84","doi-asserted-by":"publisher","first-page":"3192","DOI":"10.1039\/C6SC05720A","article-title":"ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost","volume":"8","author":"Smith","year":"2017","journal-title":"Chem. Sci."},{"key":"mlstad594abib85","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/j.cpc.2018.03.016","article-title":"DeePMD-kit: a deep learning package for many-body potential energy representation and molecular dynamics","volume":"228","author":"Wang","year":"2018","journal-title":"Comput. Phys. Commun."},{"key":"mlstad594abib86","doi-asserted-by":"publisher","first-page":"8020","DOI":"10.1021\/acs.jctc.3c00704","article-title":"Robustness of local predictions in atomistic machine learning models","volume":"19","author":"Chong","year":"2023","journal-title":"J. Chem. Theory Comput."},{"key":"mlstad594abib87","doi-asserted-by":"publisher","first-page":"1110","DOI":"10.1073\/pnas.1815117116","article-title":"Ab initio thermodynamics of liquid and solid water","volume":"116","author":"Cheng","year":"2019","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstad594abib88","doi-asserted-by":"publisher","first-page":"6158","DOI":"10.1063\/1.478522","article-title":"Toward reliable density functional methods without adjustable parameters: the PBE0 model","volume":"110","author":"Adamo","year":"1999","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib89","doi-asserted-by":"publisher","first-page":"890","DOI":"10.1103\/PhysRevLett.80.890","article-title":"Comment on \u2018generalized gradient approximation made simple\u2019","volume":"80","author":"Zhang","year":"1998","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib90","doi-asserted-by":"publisher","first-page":"6670","DOI":"10.1039\/c0cp02984j","article-title":"A thorough benchmark of density functional methods for general main group thermochemistry, kinetics and noncovalent interactions","volume":"13","author":"Goerigk","year":"2011","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstad594abib91","doi-asserted-by":"publisher","DOI":"10.1063\/1.3382344","article-title":"A consistent and accurate ab initio parametrization of density functional dispersion correction (DFT-D) for the 94 elements H-Pu","volume":"132","author":"Grimme","year":"2010","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib92","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2017"},{"key":"mlstad594abib93","article-title":"On the convergence of Adam and beyond","author":"Reddi","year":"2019"},{"key":"mlstad594abib94","doi-asserted-by":"publisher","first-page":"1482","DOI":"10.1021\/acs.chemmater.3c02726","article-title":"Mechanism of charge transport in lithium thiophosphate","volume":"36","author":"Gigli","year":"2024","journal-title":"Chem. Mater."},{"key":"mlstad594abib95","doi-asserted-by":"publisher","first-page":"3865","DOI":"10.1103\/PhysRevLett.77.3865","article-title":"Generalized gradient approximation made simple","volume":"77","author":"Perdew","year":"1996","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib96","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.100.136406","article-title":"Restoring the density-gradient expansion for exchange in solids and surfaces","volume":"100","author":"Perdew","year":"2008","journal-title":"Phys. Rev. Lett."},{"key":"mlstad594abib97","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1038\/s41524-022-00845-0","article-title":"Thermodynamics and dielectric response of BaTiO3 by data-driven modeling","volume":"8","author":"Gigli","year":"2022","journal-title":"npj Comput. Mater."},{"key":"mlstad594abib98","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2014.22","article-title":"Quantum chemistry structures and properties of 134 kilo molecules","volume":"1","author":"Ramakrishnan","year":"2014","journal-title":"Sci. Data"},{"key":"mlstad594abib99","doi-asserted-by":"publisher","first-page":"3098","DOI":"10.1103\/PhysRevA.38.3098","article-title":"Density-functional exchange-energy approximation with correct asymptotic behavior","volume":"38","author":"Becke","year":"1988","journal-title":"Phys. Rev. A"},{"key":"mlstad594abib100","doi-asserted-by":"publisher","first-page":"5648","DOI":"10.1063\/1.464913","article-title":"Density-functional thermochemistry. III. The role of exact exchange","volume":"98","author":"Becke","year":"1993","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib101","doi-asserted-by":"publisher","first-page":"3265","DOI":"10.1063\/1.447079","article-title":"Self-consistent molecular orbital methods 25. Supplementary functions forGaussian basis sets","volume":"80","author":"Frisch","year":"1984","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib102","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1103\/PhysRevB.37.785","article-title":"Development of the Colle-Salvetti correlation-energy formula into a functional of the electron density","volume":"37","author":"Lee","year":"1988","journal-title":"Phys. Rev. B"},{"key":"mlstad594abib103","doi-asserted-by":"publisher","DOI":"10.1039\/C8CP05921G","article-title":"Feature optimization for atomistic machine learning yields a data-driven construction of the periodic table of the elements","volume":"20","author":"Willatt","year":"2018","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstad594abib104","article-title":"MACE: higher order equivariant message passing neural networks for fast and accurate force fields","author":"Batatia","year":"2022"},{"key":"mlstad594abib105","article-title":"Wigner kernels: body-ordered equivariant machine learning without a basis","author":"Bigi","year":"2023"},{"key":"mlstad594abib106","article-title":"TensorNet: cartesian tensor representations for efficient learning of molecular potentials","volume":"vol 36","author":"Simeon","year":"2024"},{"key":"mlstad594abib107","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1021\/acs.jctc.9b00805","article-title":"Automated fitting of neural network potentials at coupled cluster accuracy: protonated water clusters astesting ground","volume":"16","author":"Schran","year":"2020","journal-title":"J. Chem. Theory Comput."},{"key":"mlstad594abib108","doi-asserted-by":"publisher","DOI":"10.1063\/1.5023802","article-title":"Less is more: sampling chemical space with active learning","volume":"148","author":"Smith","year":"2018","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib109","article-title":"On the uncertainty estimates of equivariant-neural-network-ensembles interatomic potentials","author":"Lu","year":"2023"},{"key":"mlstad594abib110","article-title":"Admissible scoring systems for continuous distributions","author":"Brown","year":"1974"},{"key":"mlstad594abib111","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1175\/1520-0434(2000)0152.0.CO;2","article-title":"Decomposition of the continuous ranked probability score for ensemble prediction systems","volume":"15","author":"Hersbach","year":"2000","journal-title":"Weather Forecast."},{"key":"mlstad594abib112","article-title":"Coherent energy and force uncertainty in deep learning force fields","author":"J\u00f8rgensen","year":"2023"},{"key":"mlstad594abib113","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.cpc.2018.09.020","article-title":"I-PI 2.0: a universal force engine for advanced molecular simulations","volume":"236","author":"Kapil","year":"2019","journal-title":"Comput. Phys. Commun."},{"key":"mlstad594abib114","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/S0301-0104(00)00179-8","article-title":"The radial distribution functions of water and ice from 220 to 673K and at pressures up to 400MPa","volume":"258","author":"Soper","year":"2000","journal-title":"Chem. Phys."},{"key":"mlstad594abib115","doi-asserted-by":"publisher","DOI":"10.1063\/1.4902412","article-title":"The structure of water around the compressibility minimum","volume":"141","author":"Skinner","year":"2014","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib116","doi-asserted-by":"publisher","first-page":"1578","DOI":"10.1063\/1.466584","article-title":"X-ray scattering in liquid water at pressures of up to 7.7 kbar: test of a fluctuation model","volume":"100","author":"Okhulkov","year":"1994","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib117","doi-asserted-by":"publisher","DOI":"10.1063\/1.3298879","article-title":"Heat capacity of water: a signature of nuclear quantum effects","volume":"132","author":"Vega","year":"2010","journal-title":"J. Chem. Phys."},{"key":"mlstad594abib118","doi-asserted-by":"crossref","DOI":"10.1088\/2632-2153\/ad805f","article-title":"A prediction rigidity formalism for low-cost uncertainties in trained neural networks","author":"Bigi","year":"2024"},{"key":"mlstad594abib119","article-title":"Reliable training and estimation of variance networks","volume":"vol 32","author":"Skafte","year":"2019"},{"key":"mlstad594abib120","first-page":"pp 2696","article-title":"Student-t variational autoencoder for robust density estimation","author":"Takahashi","year":"2018"},{"key":"mlstad594abib121","article-title":"Variational variance: simple,reliable,calibrated heteroscedastic noise variance parameterization","author":"Stirn","year":"2020"},{"key":"mlstad594abib122","article-title":"The uci machine learning repository","author":"Kelly"},{"key":"mlstad594abib123","author":"Kin family of datasets","year":"2024"},{"key":"mlstad594abib124","author":"Statlib archive","year":"2024"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T20:45:53Z","timestamp":1732221953000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad594a"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,5]]},"references-count":124,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,7,5]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad594a","relation":{"has-review":[{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/AD594A\/v1\/review2","asserted-by":"object"}]},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,5]]},"assertion":[{"value":"Uncertainty quantification by direct propagation of shallow ensembles","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-02-26","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-06-17","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-07-05","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}