{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T17:38:03Z","timestamp":1778693883108,"version":"3.51.4"},"reference-count":41,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T00:00:00Z","timestamp":1640131200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T00:00:00Z","timestamp":1640131200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100009708","name":"Novo Nordisk Fonden","doi-asserted-by":"crossref","award":["NNF19OC0057822"],"award-info":[{"award-number":["NNF19OC0057822"]}],"id":[{"id":"10.13039\/501100009708","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"crossref","award":["957189"],"award-info":[{"award-number":["957189"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2022,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Data-driven methods based on machine learning have the potential to accelerate computational analysis of atomic structures. In this context, reliable uncertainty estimates are important for assessing confidence in predictions and enabling decision making. However, machine learning models can produce badly calibrated uncertainty estimates and it is therefore crucial to detect and handle uncertainty carefully. In this work we extend a message passing neural network designed specifically for predicting properties of molecules and materials with a calibrated probabilistic predictive distribution. The method presented in this paper differs from previous work by considering both aleatoric and epistemic uncertainty in a unified framework, and by recalibrating the predictive distribution on unseen data. Through computer experiments, we show that our approach results in accurate models for predicting molecular formation energies with well calibrated uncertainty in and out of the training data distribution on two public molecular benchmark datasets, QM9 and PC9. The proposed method provides a general framework for training and evaluating neural network ensemble models that are able to produce accurate predictions of properties of molecules with well calibrated uncertainty estimates.<\/jats:p>","DOI":"10.1088\/2632-2153\/ac3eb3","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T04:48:32Z","timestamp":1638334112000},"page":"015012","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks"],"prefix":"10.1088","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2502-1679","authenticated-orcid":false,"given":"Jonas","family":"Busk","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4404-7276","authenticated-orcid":false,"given":"Peter","family":"Bj\u00f8rn J\u00f8rgensen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3198-5116","authenticated-orcid":false,"given":"Arghya","family":"Bhowmik","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6927-8869","authenticated-orcid":false,"given":"Mikkel N","family":"Schmidt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1966-3205","authenticated-orcid":false,"given":"Ole","family":"Winther","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1484-0284","authenticated-orcid":false,"given":"Tejs","family":"Vegge","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2021,12,22]]},"reference":[{"key":"mlstac3eb3bib1","doi-asserted-by":"publisher","first-page":"2336","DOI":"10.1021\/acs.jpclett.9b03664","article-title":"Quantum chemistry in the age of machine learning","volume":"11","author":"Dral","year":"2020","journal-title":"J. Phys. Chem. Lett."},{"key":"mlstac3eb3bib2","doi-asserted-by":"publisher","first-page":"4895","DOI":"10.1038\/s41467-020-18556-9","article-title":"Retrospective on a decade of machine learning for chemical discovery","volume":"11","author":"von Lilienfeld","year":"2020","journal-title":"Nat. Commun."},{"key":"mlstac3eb3bib3","doi-asserted-by":"publisher","first-page":"10978","DOI":"10.1039\/C7CP00375G","article-title":"Addressing uncertainty in atomistic machine learning","volume":"19","author":"Peterson","year":"2017","journal-title":"Phys. Chem. Chem. Phys."},{"key":"mlstac3eb3bib4","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ab7e1a","article-title":"Methods for comparing uncertainty quantifications for material property predictions","volume":"1","author":"Tran","year":"2020","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstac3eb3bib5","first-page":"pp 1263","article-title":"Neural message passing for quantum chemistry","author":"Gilmer","year":"2017"},{"key":"mlstac3eb3bib6","first-page":"pp 2796","article-title":"Accurate uncertainties for deep learning using calibrated regression","author":"Kuleshov","year":"2018"},{"key":"mlstac3eb3bib7","first-page":"pp 5897","article-title":"Distribution calibration for regression","author":"Song","year":"2019"},{"key":"mlstac3eb3bib8","article-title":"Evaluating and calibrating uncertainty prediction in regression tasks","author":"Levi","year":"2020"},{"key":"mlstac3eb3bib9","article-title":"On the quality of uncertainty estimates from neural network potential ensembles","author":"Kahle","year":"2021"},{"key":"mlstac3eb3bib10","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1021\/acscentsci.7b00572","article-title":"Automatic chemical design using a data-driven continuous representation of molecules","volume":"4","author":"G\u00f3mez-Bombarelli","year":"2018","journal-title":"ACS Cent. Sci."},{"key":"mlstac3eb3bib11","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1007\/s10822-013-9672-4","article-title":"Estimation of the size of drug-like chemical space based on GDB-17 data","volume":"27","author":"Polishchuk","year":"2013","journal-title":"J. Comput.-Aided Mol. Des."},{"key":"mlstac3eb3bib12","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1021\/acsmaterialslett.0c00028","article-title":"Molecular engineering strategies for symmetric aqueous organic redox flow batteries","volume":"2","author":"Fornari","year":"2020","journal-title":"ACS Mater. Lett."},{"key":"mlstac3eb3bib13","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1002\/batt.201900152","article-title":"Autonomous discovery of materials for intercalation electrodes","volume":"3","author":"B\u00f6lle","year":"2020","journal-title":"Batteries Supercaps"},{"key":"mlstac3eb3bib14","doi-asserted-by":"publisher","first-page":"1963","DOI":"10.1039\/D0RE00232A","article-title":"Iterative experimental design based on active machine learning reduces the experimental burden associated with reaction screening","volume":"5","author":"Eyke","year":"2020","journal-title":"React. Chem. Eng."},{"key":"mlstac3eb3bib15","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1016\/j.cels.2020.09.007","article-title":"Leveraging uncertainty in machine learning accelerates biological discovery and design","volume":"11","author":"Hie","year":"2020","journal-title":"Cell Syst."},{"key":"mlstac3eb3bib16","doi-asserted-by":"publisher","first-page":"1356","DOI":"10.1021\/acscentsci.1c00546","article-title":"Evidential deep learning for guided molecular property prediction and discovery","volume":"7","author":"Soleimany","year":"2021","journal-title":"ACS Cent. Sci."},{"key":"mlstac3eb3bib17","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","article-title":"Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods","volume":"110","author":"H\u00fcllermeier","year":"2021","journal-title":"Mach. Learn."},{"key":"mlstac3eb3bib18","first-page":"pp 5580","article-title":"What uncertainties do we need in bayesian deep learning for computer vision?","author":"Kendall","year":"2017"},{"key":"mlstac3eb3bib19","doi-asserted-by":"publisher","first-page":"2697","DOI":"10.1021\/acs.jcim.9b00975","article-title":"Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction","volume":"60","author":"Scalia","year":"2020","journal-title":"J. Chem. Inf. Model."},{"key":"mlstac3eb3bib20","first-page":"pp 6402","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"vol 30","author":"Lakshminarayanan","year":"2017"},{"key":"mlstac3eb3bib21","first-page":"pp 1050","article-title":"Dropout as a Bayesian approximation: representing model uncertainty in deep learning","author":"Gal","year":"2016"},{"key":"mlstac3eb3bib22","doi-asserted-by":"publisher","first-page":"3770","DOI":"10.1021\/acs.jcim.0c00502","article-title":"Uncertainty quantification using neural networks for molecular property prediction","volume":"60","author":"Hirschfeld","year":"2020","journal-title":"J. Chem. Inf. Model."},{"key":"mlstac3eb3bib23","doi-asserted-by":"crossref","DOI":"10.1080\/17460441.2021.1925247","article-title":"Assigning confidence to molecular property prediction","author":"Nigam","year":"2021"},{"key":"mlstac3eb3bib24","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1021\/acs.jctc.8b00959","article-title":"Fast and accurate uncertainty estimation in chemical machine learning","volume":"15","author":"Musil","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstac3eb3bib25","doi-asserted-by":"publisher","DOI":"10.1063\/5.0036522","article-title":"Uncertainty estimation for molecular dynamics and sampling","volume":"154","author":"Imbalzano","year":"2021","journal-title":"J. Chem. Phys."},{"key":"mlstac3eb3bib26","article-title":"Neural message passing with edge updates for predicting properties of molecules and materials","author":"J\u00f8rgensen","year":"2018"},{"key":"mlstac3eb3bib27","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2014.22","article-title":"Quantum chemistry structures and properties of 134 kilo molecules","volume":"1","author":"Ramakrishnan","year":"2014","journal-title":"Sci. Data"},{"key":"mlstac3eb3bib28","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/s13321-019-0391-2","article-title":"Dataset\u2019s chemical diversity limits the generalizability of machine learning predictions","volume":"11","author":"Glavatskikh","year":"2019","journal-title":"J. Cheminform."},{"key":"mlstac3eb3bib29","first-page":"pp 991","article-title":"SchNet: a continuous-filter convolutional neural network for modeling quantum interactions","volume":"vol 30","author":"Sch\u00fctt","year":"2017"},{"key":"mlstac3eb3bib30","first-page":"pp 55","article-title":"Estimating the mean and variance of the target probability distribution","volume":"vol 1","author":"Nix","year":"1994"},{"key":"mlstac3eb3bib31","first-page":"pp 14927","article-title":"Deep evidential regression","volume":"vol 33","author":"Amini","year":"2020"},{"key":"mlstac3eb3bib32","article-title":"Reliable training and estimation of variance networks","author":"Detlefsen","year":"2019"},{"key":"mlstac3eb3bib33","doi-asserted-by":"publisher","first-page":"12","DOI":"10.2307\/2987588","article-title":"The comparison and evaluation of forecasters","volume":"32","author":"DeGroot","year":"1983","journal-title":"J. R. Stat. Soc. D"},{"key":"mlstac3eb3bib34","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1080\/01621459.1982.10477856","article-title":"The well-calibrated bayesian","volume":"77","author":"Dawid","year":"1982","journal-title":"J. Am. Stat. Assoc."},{"key":"mlstac3eb3bib35","first-page":"pp 1321","article-title":"On calibration of modern neural networks","volume":"vol 70","author":"Guo","year":"2017"},{"key":"mlstac3eb3bib36","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"mlstac3eb3bib37","doi-asserted-by":"publisher","first-page":"D1388","DOI":"10.1093\/nar\/gkaa971","article-title":"PubChem in 2021: new data content and improved web interfaces","volume":"49","author":"Kim","year":"2020","journal-title":"Nucleic Acids Res."},{"key":"mlstac3eb3bib38","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13321-015-0068-4","article-title":"The IUPAC International Chemical Identifier","volume":"7","author":"Heller","year":"2015","journal-title":"J. Cheminform."},{"key":"mlstac3eb3bib39","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2019"},{"key":"mlstac3eb3bib40","first-page":"pp 8617","article-title":"Randomized prior functions for deep reinforcement learning","volume":"vol 31","author":"Osband","year":"2018"},{"key":"mlstac3eb3bib41","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1016\/j.ensm.2019.06.011","article-title":"A perspective on inverse design of battery interphases using multi-scale modelling, experiments and generative deep learning","volume":"21","author":"Bhowmik","year":"2019","journal-title":"Energy Storage Mater."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,22]],"date-time":"2021-12-22T06:18:37Z","timestamp":1640153917000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac3eb3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,22]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12,22]]},"published-print":{"date-parts":[[2022,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ac3eb3","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,22]]},"assertion":[{"value":"Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2021-11-03","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-11-30","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-12-22","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}