{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T16:13:04Z","timestamp":1775664784420,"version":"3.50.1"},"reference-count":57,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T00:00:00Z","timestamp":1671580800000},"content-version":"vor","delay-in-days":20,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T00:00:00Z","timestamp":1671580800000},"content-version":"tdm","delay-in-days":20,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"crossref","award":["DE-SC0019441"],"award-info":[{"award-number":["DE-SC0019441"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2022,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Uncertainty quantification (UQ) is important to machine learning (ML) force fields to assess the level of confidence during prediction, as ML models are not inherently physical and can therefore yield catastrophically incorrect predictions. Established <jats:italic>a-posteriori<\/jats:italic> UQ methods, including ensemble methods, the dropout method, the delta method, and various heuristic distance metrics, have limitations such as being computationally challenging for large models due to model re-training. In addition, the uncertainty estimates are often not rigorously calibrated. In this work, we propose combining the distribution-free UQ method, known as conformal prediction (CP), with the distances in the neural network\u2019s latent space to estimate the uncertainty of energies predicted by neural network force fields. We evaluate this method (CP+latent) along with other UQ methods on two essential aspects, calibration, and sharpness, and find this method to be both calibrated and sharp under the assumption of independent and identically-distributed (i.i.d.) data. We show that the method is relatively insensitive to hyperparameters selected, and test the limitations of the method when the i.i.d. assumption is violated. Finally, we demonstrate that this method can be readily applied to trained neural network force fields with traditional and graph neural network architectures to obtain estimates of uncertainty with low computational costs on a training dataset of 1 million images to showcase its scalability and portability. Incorporating the CP method with latent distances offers a calibrated, sharp and efficient strategy to estimate the uncertainty of neural network force fields. In addition, the CP approach can also function as a promising strategy for calibrating uncertainty estimated by other approaches.<\/jats:p>","DOI":"10.1088\/2632-2153\/aca7b1","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T22:41:35Z","timestamp":1669848095000},"page":"045028","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials"],"prefix":"10.1088","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3648-7749","authenticated-orcid":false,"given":"Yuge","family":"Hu","sequence":"first","affiliation":[]},{"given":"Joseph","family":"Musielewicz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9401-4918","authenticated-orcid":true,"given":"Zachary W","family":"Ulissi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8311-9581","authenticated-orcid":true,"given":"Andrew J","family":"Medford","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2022,12,21]]},"reference":[{"key":"mlstaca7b1bib1","doi-asserted-by":"publisher","first-page":"17811","DOI":"10.1021\/acs.jpcc.0c04225","article-title":"SingleNN: modified Behler-Parrinello neural network with shared weights for atomistic simulations with transferability","volume":"124","author":"Liu","year":"2020","journal-title":"J. Phys. Chem. C"},{"key":"mlstaca7b1bib2","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.87.184115","article-title":"On representing chemical environments","volume":"87","author":"Bart\u00f3k","year":"2013","journal-title":"Phys. Rev. B"},{"key":"mlstaca7b1bib3","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.98.146401","article-title":"Generalized neural-network representation of high-dimensional potential-energy surfaces","volume":"98","author":"Behler","year":"2007","journal-title":"Phys. Rev. Lett."},{"key":"mlstaca7b1bib4","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1016\/j.cattod.2018.03.045","article-title":"Modeling palladium surfaces with density functional theory, neural networks and molecular dynamics","volume":"312","author":"Gao","year":"2018","journal-title":"Catal. Today"},{"key":"mlstaca7b1bib5","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1080\/08927022.2016.1274984","article-title":"Neural network predictions of oxygen interactions on a dynamic Pd surface","volume":"43","author":"Boes","year":"4 2017","journal-title":"Mol. Simul."},{"key":"mlstaca7b1bib6","doi-asserted-by":"publisher","first-page":"3192","DOI":"10.1039\/C6SC05720A","article-title":"ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost","volume":"8","author":"Smith","year":"2017","journal-title":"Chem. Sci."},{"key":"mlstaca7b1bib7","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1126\/sciadv.1603015","article-title":"Machine learning of accurate energy-conserving molecular force fields","volume":"3","author":"Chmiela","year":"2017","journal-title":"Sci. Adv."},{"key":"mlstaca7b1bib8","doi-asserted-by":"publisher","DOI":"10.1063\/1.5019779","article-title":"SchNet\u2014a deep learning architecture for molecules and materials","volume":"148","author":"Sch\u00fctt","year":"2018","journal-title":"J. Chem. Phys."},{"key":"mlstaca7b1bib9","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.83.153101","article-title":"High-dimensional neural-network potentials for multicomponent systems: applications to zinc oxide","volume":"83","author":"Artrith","year":"2011","journal-title":"Phy. Rev. B"},{"key":"mlstaca7b1bib10","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1080\/08927022.2017.1420185","article-title":"A density functional theory parameterised neural network model of zirconia","volume":"44","author":"Wang","year":"5 2018","journal-title":"Mol. Simul."},{"key":"mlstaca7b1bib11","doi-asserted-by":"publisher","first-page":"1827","DOI":"10.1021\/acs.jctc.8b00770","article-title":"Library-based lammps implementation of high-dimensional neural network potentials","volume":"15","author":"Singraber","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaca7b1bib12","doi-asserted-by":"publisher","first-page":"28704","DOI":"10.1039\/C6CP05711J","article-title":"Neural network molecular dynamics simulations of solid\u2013liquid interfaces: water at low-index copper surfaces","volume":"18","author":"Natarajan","year":"2016","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstaca7b1bib13","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1016\/j.cpc.2016.05.010","article-title":"Amp: a modular approach to machine learning in atomistic simulations","volume":"207","author":"Khorshidi","year":"10 2016","journal-title":"Comput. Phys. Commun."},{"key":"mlstaca7b1bib14","doi-asserted-by":"publisher","first-page":"6059","DOI":"10.1021\/acscatal.0c04525","article-title":"Open catalyst 2020 (OC20) dataset and community challenges","volume":"11","author":"Chanussot","year":"5 2021","journal-title":"ACS Catal."},{"key":"mlstaca7b1bib15","article-title":"Rotation Invariant graph neural networks using spin convolutions","author":"Shuaibi","year":"2021"},{"key":"mlstaca7b1bib16","doi-asserted-by":"publisher","first-page":"03LT01","DOI":"10.1088\/2632-2153\/ac8fe0","article-title":"FINETUNA: fine-tuning accelerated molecular simulations","volume":"3","author":"Musielewicz","year":"2022","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstaca7b1bib17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1140\/epjb\/s10051-021-00156-1","article-title":"Machine learning potentials for extended systems: a perspective","volume":"94","author":"Behler","year":"2021","journal-title":"Eur. Phys. J. B"},{"key":"mlstaca7b1bib18","doi-asserted-by":"publisher","DOI":"10.1063\/1.5142636","article-title":"Approaches for machine learning intermolecular interaction energies and application to energy components from symmetry adapted perturbation theory","volume":"152","author":"Metcalf","year":"2020","journal-title":"J. Chem. Phys."},{"key":"mlstaca7b1bib19","doi-asserted-by":"publisher","DOI":"10.1063\/5.0042989","article-title":"CLIFF: a component-based, machine-learned, intermolecular force field","volume":"154","author":"Schriber","year":"2021","journal-title":"J. Chem. Phys."},{"key":"mlstaca7b1bib20","doi-asserted-by":"publisher","first-page":"2903","DOI":"10.1038\/s41467-019-10827-4","article-title":"Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning","volume":"10","author":"Smith","year":"2019","journal-title":"Nat. Commun."},{"key":"mlstaca7b1bib21","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1021\/acs.jctc.9b00805","article-title":"Automated fitting of neural network potentials at coupled cluster accuracy: protonated water clusters as testing ground","volume":"16","author":"Schran","year":"2020","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaca7b1bib22","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.106.L041105","article-title":"High pressure hydrogen by machine learning and quantum Monte Carlo","volume":"106","author":"Tirelli","year":"12 2021","journal-title":"Phys. Rev. B"},{"key":"mlstaca7b1bib23","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1016\/j.cpc.2019.02.007","article-title":"sGDML: constructing accurate and data efficient molecular force fields using machine learning","volume":"240","author":"Chmiela","year":"2019","journal-title":"Comput. Phys. Commun."},{"key":"mlstaca7b1bib24","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1002\/qua.24927","article-title":"Gaussian approximation potentials: a brief tutorial introduction","volume":"115","author":"Bart\u00f5k","year":"2015","journal-title":"Int. J. Quantum Chem."},{"key":"mlstaca7b1bib25","doi-asserted-by":"publisher","first-page":"1032","DOI":"10.1002\/qua.24890","article-title":"Constructing high-dimensional neural network potentials: a tutorial review","volume":"115","author":"Behler","year":"2015","journal-title":"Int. J. Quantum Chem."},{"key":"mlstaca7b1bib26","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.120.143001","article-title":"Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics","volume":"120","author":"Zhang","year":"2018","journal-title":"Phys. Rev. Lett."},{"key":"mlstaca7b1bib27","doi-asserted-by":"publisher","first-page":"7911","DOI":"10.1021\/acs.jpclett.2c02100","article-title":"A universal framework for featurization of atomistic systems","volume":"13","author":"Lei","year":"2022","journal-title":"J. Phys. Chem. Lett."},{"key":"mlstaca7b1bib28","doi-asserted-by":"publisher","first-page":"3678","DOI":"10.1021\/acs.jctc.9b00181","article-title":"PhysNet: a neural network for predicting energies, forces, dipole moments and partial charges","volume":"15","author":"Unke","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaca7b1bib29","article-title":"Fast and uncertainty-aware directional message passing for non-equilibrium molecules","author":"Gasteiger","year":"2022"},{"key":"mlstaca7b1bib30","article-title":"GemNet: universal directional graph neural networks for molecules","author":"Gasteiger","year":"2022"},{"key":"mlstaca7b1bib31","doi-asserted-by":"publisher","first-page":"14396","DOI":"10.1039\/D1SC03564A","article-title":"Choosing the right molecular machine learning potential","volume":"12","author":"Pinheiro","year":"2021","journal-title":"Chem. Sci."},{"key":"mlstaca7b1bib32","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ab7e1a","article-title":"Methods for comparing uncertainty quantifications for material property predictions","volume":"1","author":"Tran","year":"2020","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstaca7b1bib33","doi-asserted-by":"publisher","first-page":"10978","DOI":"10.1039\/C7CP00375G","article-title":"Addressing uncertainty in atomistic machine learning","volume":"19","author":"Peterson","year":"2017","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstaca7b1bib34","doi-asserted-by":"publisher","DOI":"10.1063\/5.0084302","article-title":"The long road to calibrated prediction uncertainty in computational chemistry","volume":"156","author":"Pernot","year":"2022","journal-title":"J. Chem. Phys."},{"key":"mlstaca7b1bib35","doi-asserted-by":"publisher","first-page":"2392","DOI":"10.1021\/acs.jcim.8b00386","article-title":"Dynamic workflows for routine materials discovery in surface science","volume":"58","author":"Tran","year":"2018","journal-title":"J. Chem. Inf. Model."},{"key":"mlstaca7b1bib36","doi-asserted-by":"publisher","first-page":"1257","DOI":"10.1038\/s41467-021-21376-0","article-title":"Automated discovery of a robust interatomic potential for aluminum","volume":"12","author":"Smith","year":"2021","journal-title":"Nat. Commun."},{"key":"mlstaca7b1bib37","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1038\/s41524-020-00390-8","article-title":"Uncertainty quantification in molecular simulations with dropout neural network potentials","volume":"6","author":"Wen","year":"2020","journal-title":"npj Comput. Mater."},{"key":"mlstaca7b1bib38","doi-asserted-by":"publisher","DOI":"10.1002\/aic.17516","article-title":"Uncertainty quantification in machine learning and nonlinear least squares regression models","volume":"68","author":"Zhan","year":"2021","journal-title":"AIChE J."},{"key":"mlstaca7b1bib39","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1016\/j.neunet.2021.10.014","article-title":"Epistemic uncertainty quantification in deep learning classification by the Delta method","volume":"145","author":"Nilsen","year":"2022","journal-title":"Neural Netw."},{"key":"mlstaca7b1bib40","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1080\/00223131.2015.1034216","article-title":"Confidence interval estimation by bootstrap method for uncertainty quantification using random sampling method","volume":"52","author":"Endo","year":"2015","journal-title":"J. Nucl.Sci.Technol."},{"key":"mlstaca7b1bib41","doi-asserted-by":"publisher","first-page":"3700","DOI":"10.21105\/joss.03700","article-title":"UnlockNN: uncertainty quantification for neural network models of chemical systems","volume":"7","author":"Moriarty","year":"2022","journal-title":"J. Open Source Softw."},{"key":"mlstaca7b1bib42","doi-asserted-by":"publisher","first-page":"12078","DOI":"10.1609\/aaai.v35i13.17434","article-title":"Uncertainty quantification in cnn through the bootstrap of convex neural networks","volume":"vol 35 pp","author":"Du","year":"2021"},{"key":"mlstaca7b1bib43","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/s41524-022-00794-8","article-title":"Calibrated bootstrap for uncertainty quantification in regression models","volume":"5","author":"Palmer","year":"2022","journal-title":"npj Comput. Mater."},{"key":"mlstaca7b1bib44","doi-asserted-by":"publisher","first-page":"7913","DOI":"10.1039\/C9SC02298H","article-title":"A quantitative uncertainty metric controls error in neural network-driven chemical discovery","volume":"10","author":"Janet","year":"2019","journal-title":"Chem. Sci."},{"key":"mlstaca7b1bib45","doi-asserted-by":"publisher","first-page":"511","DOI":"10.1021\/acs.jpcc.6b10908","article-title":"Machine learning force fields: construction, validation and outlook","volume":"121","author":"Botu","year":"2017","journal-title":"J. Phys. Chem. C"},{"key":"mlstaca7b1bib46","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.105151","article-title":"Ensemble deep learning: a review","volume":"115","author":"Ganaie","year":"2022","journal-title":"Eng. Appl. Artif. Intell."},{"key":"mlstaca7b1bib47","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1021\/acs.jcim.8b00597","article-title":"Molecular Similarity-based domain applicability metric efficiently identifies out-of-domain compounds","volume":"59","author":"Liu","year":"2019","journal-title":"J. Chem. Inf. Model."},{"key":"mlstaca7b1bib48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/sdata.2014.22","article-title":"Quantum chemistry structures and properties of 134 kilo molecules","volume":"1","author":"Ramakrishnan","year":"2014","journal-title":"Sci. Data"},{"key":"mlstaca7b1bib49","article-title":"GemNet-OC: developing graph neural networks for large and diverse molecular simulation datasets","author":"Gasteiger","year":"2022"},{"key":"mlstaca7b1bib50","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1038\/s41929-018-0142-1","article-title":"Active learning across intermetallics to guide discovery of electrocatalysts for CO2 reduction and H2 evolution","volume":"1","author":"Tran","year":"2018","journal-title":"Nat. Catal."},{"key":"mlstaca7b1bib51","first-page":"pp 4369","article-title":"Accurate uncertainties for deep learning using calibrated regression","volume":"vol 6","author":"Kuleshov","year":"2018"},{"key":"mlstaca7b1bib52","first-page":"1","article-title":"A gentle introduction to conformal prediction and distribution-free uncertainty quantification","author":"Angelopoulos","year":"2021"},{"key":"mlstaca7b1bib53","article-title":"Conformalized quantile regression","author":"Romano","year":"2019"},{"key":"mlstaca7b1bib54","doi-asserted-by":"publisher","first-page":"1094","DOI":"10.1080\/01621459.2017.1307116","article-title":"Distribution-free predictive inference for regression","volume":"113","author":"Lei","year":"2016","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstaca7b1bib55","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1038\/s41524-021-00508-6","article-title":"A systematic approach to generating accurate neural network potentials: the case of carbon","volume":"7","author":"Shaidu","year":"2021","journal-title":"npj Comput. Mater."},{"key":"mlstaca7b1bib56","doi-asserted-by":"publisher","first-page":"819","DOI":"10.1039\/C8ME00012C","article-title":"Can machine learning identify the next high-temperature superconductor? examining extrapolation performance for materials discovery","volume":"3","author":"Meredig","year":"2018","journal-title":"Mol. Syst. Des. Eng."},{"key":"mlstaca7b1bib57","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1016\/0377-2217(80)90107-1","article-title":"Building a balanced k-d tree in O(kn log n) time","volume":"4","author":"Camerini","year":"1980","journal-title":"Eur. J. Oper. Res."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T09:59:06Z","timestamp":1671616746000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aca7b1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,1]]},"references-count":57,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12,21]]},"published-print":{"date-parts":[[2022,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/aca7b1","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,1]]},"assertion":[{"value":"Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2022 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2022-08-22","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-11-30","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2022-12-21","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}