{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T19:59:25Z","timestamp":1775764765289,"version":"3.50.1"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009708","name":"Novo Nordisk Fonden","doi-asserted-by":"crossref","award":["NNF20OC0064104"],"award-info":[{"award-number":["NNF20OC0064104"]}],"id":[{"id":"10.13039\/501100009708","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100006151","name":"Basic Energy Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006151","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001734","name":"Copenhagen University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001734","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    With the increasingly more important role of machine learning (ML) models in chemical research, the need for putting a level of confidence to the model predictions naturally arises. Several methods for obtaining uncertainty estimates have been proposed in recent years but consensus on the evaluation of these have yet to be established and different studies on uncertainties generally uses different metrics to evaluate them. We compare three of the most popular validation metrics (Spearman\u2019s rank correlation coefficient, the negative log likelihood (NLL) and the miscalibration area) to the error-based calibration introduced by Levi et al. (\n                    <jats:italic>Sensors<\/jats:italic>\n                    <jats:bold>2022<\/jats:bold>\n                    ,\n                    <jats:italic>22<\/jats:italic>\n                    , 5540). Importantly, metrics such as the negative log likelihood (NLL) and Spearman\u2019s rank correlation coefficient bear little information in themselves. We therefore introduce reference values obtained through errors simulated directly from the uncertainty distribution. The different metrics target different properties and we show how to interpret them, but we generally find the best overall validation to be done based on the error-based calibration plot introduced by Levi et al. Finally, we illustrate the sensitivity of ranking-based methods (e.g. Spearman\u2019s rank correlation coefficient) towards test set design by using the same toy model ferent test sets and obtaining vastly different metrics (0.05 vs. 0.65).\n                  <\/jats:p>","DOI":"10.1186\/s13321-023-00790-0","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T10:02:13Z","timestamp":1702893733000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets"],"prefix":"10.1186","volume":"15","author":[{"given":"Maria H.","family":"Rasmussen","sequence":"first","affiliation":[]},{"given":"Chenru","family":"Duan","sequence":"additional","affiliation":[]},{"given":"Heather J.","family":"Kulik","sequence":"additional","affiliation":[]},{"given":"Jan H.","family":"Jensen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,12,18]]},"reference":[{"issue":"34","key":"790_CR1","doi-asserted-by":"publisher","first-page":"7913","DOI":"10.1039\/c9sc02298h","volume":"10","author":"Jon Paul Janet","year":"2019","unstructured":"Janet Jon Paul, Duan Chenru, Yang Tzuhsiung, Nandy Aditya, Kulik Heather J (2019) A quantitative uncertainty metric controls error in neural network-driven chemical discovery. Chem Sci 10(34):7913\u20137922. https:\/\/doi.org\/10.1039\/c9sc02298h","journal-title":"Chem Sci"},{"issue":"8","key":"790_CR2","doi-asserted-by":"publisher","first-page":"3846","DOI":"10.1021\/acs.jcim.1c00670","volume":"61","author":"Michael Tynes","year":"2021","unstructured":"Tynes Michael, Gao Wenhao, Burrill Daniel J, Batista Enrique R, Perez Danny, Yang Ping, Lubbers Nicholas (2021) Pairwise difference regression: a machine learning meta-algorithm for improved prediction and uncertainty quantification in chemical search. J Chem Inf Model 61(8):3846\u20133857. https:\/\/doi.org\/10.1021\/acs.jcim.1c00670","journal-title":"J Chem Inf Model"},{"issue":"8","key":"790_CR3","doi-asserted-by":"publisher","first-page":"3770","DOI":"10.1021\/acs.jcim.0c00502","volume":"60","author":"Lior Hirschfeld","year":"2020","unstructured":"Hirschfeld Lior, Swanson Kyle, Yang Kevin, Barzilay Regina, Coley Connor W (2020) Uncertainty quantification using neural networks for molecular property prediction. J Chem Inf Model 60(8):3770\u20133780. https:\/\/doi.org\/10.1021\/acs.jcim.0c00502","journal-title":"J Chem Inf Model"},{"issue":"6","key":"790_CR4","doi-asserted-by":"publisher","first-page":"2697","DOI":"10.1021\/acs.jcim.9b00975","volume":"60","author":"Gabriele Scalia","year":"2020","unstructured":"Scalia Gabriele, Grambow Colin A, Pernici Barbara, Li Yi-Pei, Green William H (2020) Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. J Chem Inf Model 60(6):2697\u20132717. https:\/\/doi.org\/10.1021\/acs.jcim.9b00975","journal-title":"J Chem Inf Model"},{"key":"790_CR5","doi-asserted-by":"crossref","unstructured":"Pernot Pascal (2022) Prediction uncertainty validation for computational chemists. arXiv:2204.13477. [physics.chem-ph]","DOI":"10.1063\/5.0109572"},{"issue":"1","key":"790_CR6","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac3eb3","volume":"3","author":"Jonas Busk","year":"2021","unstructured":"Busk Jonas, J\u00f8rgensen Peter Bj\u00f8rn, Bhowmik Arghya, Schmidt Mikkel N, Winther Ole, Vegge Tejs (2021) Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks. Mach Learn Sci Technol 3(1):015012. https:\/\/doi.org\/10.1088\/2632-2153\/ac3eb3","journal-title":"Mach Learn Sci Technol"},{"issue":"8","key":"790_CR7","doi-asserted-by":"publisher","first-page":"1356","DOI":"10.1021\/acscentsci.1c00546","volume":"7","author":"Ava P Soleimany","year":"2021","unstructured":"Soleimany Ava P, Amini Alexander, Goldman Samuel, Rus Daniela, Bhatia Sangeeta N, Coley Connor W (2021) Evidential deep learning for guided molecular property prediction and discovery. ACS Cent Sci 7(8):1356\u20131367. https:\/\/doi.org\/10.1021\/acscentsci.1c00546","journal-title":"ACS Cent Sci"},{"issue":"3","key":"790_CR8","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1039\/D2DD00146B","volume":"2","author":"Gary Tom","year":"2023","unstructured":"Tom Gary, Hickman Riley J, Zinzuwadia Aniket, Mohajeri Afshan, Sanchez-Lengeling Benjamin, Aspuru-Guzik Al\u00e1n (2023) Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS. Dig Discov 2(3):759\u2013774. https:\/\/doi.org\/10.1039\/D2DD00146B","journal-title":"Dig Discov"},{"key":"790_CR9","doi-asserted-by":"crossref","unstructured":"Varivoda D, Dong R, Omee SS, Hu J (2023) Materials property prediction with uncertainty quantification: a benchmark study. Appl Phys Rev. DOIurlhttps:\/\/doi.org\/10.1063\/5.0133528","DOI":"10.1063\/5.0133528"},{"key":"790_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106180","volume":"123","author":"Wentao Fan","year":"2023","unstructured":"Fan Wentao, Zeng Lidan, Wang Tian (2023) Uncertainty quantification in molecular property prediction through spherical mixture density networks. Eng Appl Artif Intell 123:106180. https:\/\/doi.org\/10.1016\/j.engappai.2023.106180","journal-title":"Eng Appl Artif Intell"},{"issue":"1","key":"790_CR11","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1186\/s13321-023-00709-9","volume":"15","author":"Thomas-Martin Dutschmann","year":"2023","unstructured":"Dutschmann Thomas-Martin, Kinzel Lennart, Ter Laak Antonius, Baumann Knut (2023) Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation. J Cheminformatics 15(1):49. https:\/\/doi.org\/10.1186\/s13321-023-00709-9","journal-title":"J Cheminformatics"},{"key":"790_CR12","doi-asserted-by":"publisher","DOI":"10.3390\/s22155540","author":"Levi Dan","year":"2022","unstructured":"Dan Levi, Liran Gispan, Niv Giladi, Ethan Fetaya (2022) Evaluating and calibrating uncertainty prediction in regression tasks. Sensors. https:\/\/doi.org\/10.3390\/s22155540","journal-title":"Sensors"},{"key":"790_CR13","unstructured":"Alexander A, Wilko S, Ava S, Daniela R (2020) Deep evidential regression. In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., pp. 14927\u201314937. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/aab085461de182608ee9f607f3f7d18f-Paper.pdf"},{"issue":"5","key":"790_CR14","doi-asserted-by":"publisher","first-page":"868","DOI":"10.1021\/ci990307l","volume":"39","author":"Scott A Wildman","year":"1999","unstructured":"Wildman Scott A, Crippen Gordon M (1999) Prediction of Physicochemical Parameters by Atomic Contributions. J Chem Inf Comput Sci 39(5):868\u2013873. https:\/\/doi.org\/10.1021\/ci990307l","journal-title":"J Chem Inf Comput Sci"},{"key":"790_CR15","doi-asserted-by":"publisher","unstructured":"Rasmussen MH, Christensen DS, Jensen JH (2023) Do machines dream of atoms? Crippen\u2019s logP as a quantitative molecular benchmark for explainable AI heatmaps. SciPost Chem https:\/\/doi.org\/10.21468\/scipostchem.2.1.002","DOI":"10.21468\/scipostchem.2.1.002"},{"issue":"39","key":"790_CR16","doi-asserted-by":"publisher","first-page":"13021","DOI":"10.1039\/d1sc03701c","volume":"12","author":"Duan Chenru","year":"2021","unstructured":"Chenru Duan, Shuxin Chen, Taylor Michael G, Fang Liu, Kulik Heather J (2021) Machine learning to tame divergent density functional approximations: a new path to consensus materials design principles. Chem Sci 12(39):13021\u201313036. https:\/\/doi.org\/10.1039\/d1sc03701c","journal-title":"Chem Sci"},{"issue":"4","key":"790_CR17","doi-asserted-by":"publisher","first-page":"1152","DOI":"10.1039\/d1sc05677h","volume":"13","author":"P Greenman Kevin","year":"2022","unstructured":"Greenman Kevin P, Green William H, Rafael G\u00f3mez-Bombarelli (2022) Multi-fidelity prediction of molecular optical peaks with deep learning. Chem Sci 13(4):1152\u20131162. https:\/\/doi.org\/10.1039\/d1sc05677h","journal-title":"Chem Sci"},{"key":"790_CR18","unstructured":"Pernot Pascal. \u201cConfidence curves for UQ validation: probabilistic reference vs. oracle\u201d. 2022. arXiv: 2206.15272 [physics.data-an]"},{"issue":"3","key":"790_CR19","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1214\/ss\/1032280214","volume":"11","author":"Thomas J DiCiccio","year":"1996","unstructured":"DiCiccio Thomas J, Efron Bradley (1996) Bootstrap Confidence Intervals. Stat Sci 11(3):189\u2013212","journal-title":"Stat Sci"},{"key":"790_CR20","doi-asserted-by":"publisher","unstructured":"Virtanen Pauli, Gommers Ralf, Oliphant Travis E, Haberland Matt, Reddy Tyler, Cournapeau David, Burovski Evgeni, Peterson Pearu, WarrenWeckesser Jonathan Bright, van derWalt St\u00e9fan J, Brett Matthew, Joshua Wilson K, Millman Jarrod, Mayorov Nikolay, Nelson Andrew R. J, Jones Eric, Kern Robert, Eric Larson CJ, Carey \u0130lhan Polat, Feng Yu, Moore Eric W, VanderPlas Jake, Laxalde Denis, Perktold Josef, Cimrman Robert, Ian Henriksen EA, Quintero Charles R, Harris Anne M, Archibald Ant\u00f4nio H, Ribeiro Fabian Pedregosa, van Mulbregt Paul, SciPy 1.0 Contributors, (2020) SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods 17:261\u2013272. https:\/\/doi.org\/10.1038\/s41592-019-0686-2","DOI":"10.1038\/s41592-019-0686-2"},{"issue":"1","key":"790_CR21","doi-asserted-by":"publisher","first-page":"972","DOI":"10.1080\/14686996.2017.1401424","volume":"18","author":"Xiufeng Yang","year":"2017","unstructured":"Yang Xiufeng, Zhang Jinzhe, Yoshizoe Kazuki, Terayama Kei, Tsuda Koji (2017) ChemTS: an efficient python library for de novo molecular generation. Sci Technol Adv Mater 18(1):972\u2013976. https:\/\/doi.org\/10.1080\/14686996.2017.1401424","journal-title":"Sci Technol Adv Mater"},{"issue":"2","key":"790_CR22","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1021\/acscentsci.7b00572","volume":"4","author":"Rafael G\u00f3mez-Bombarelli","year":"2018","unstructured":"G\u00f3mez-Bombarelli Rafael, NWei Jennifer, Duvenaud David, Hern\u00e1ndez-Lobato Jos\u00e9 Miguel, S\u00e1nchez-Lengeling Benjam\u00edn, Sheberla Dennis, Aguilera-Iparraguirre Jorge, Hirzel Timothy D, Adams Ryan P, Aspuru-Guzik Al\u00e1n (2018) Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent Sci 4(2):268\u2013276. https:\/\/doi.org\/10.1021\/acscentsci.7b00572","journal-title":"ACS Cent Sci"},{"key":"790_CR23","unstructured":"You Jiaxuan, Liu Bowen, Ying Rex, Pande Vijay, Leskovec Jure \u201cGraph convolutional policy network for goal-directed molecular graph generation\u201d. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS\u201918. Montr\u00e9al, Canada: Curran Associates Inc., Dec. 2018, pp. 6412-6422"},{"issue":"12","key":"790_CR24","doi-asserted-by":"publisher","first-page":"3567","DOI":"10.1039\/c8sc05372c","volume":"10","author":"Jan H Jensen","year":"2019","unstructured":"Jensen Jan H (2019) A graph-based genetic algorithm and generative model\/Monte Carlo tree search for the exploration of chemical space. Chem Sci 10(12):3567\u20133572. https:\/\/doi.org\/10.1039\/c8sc05372c","journal-title":"Chem Sci"},{"issue":"46","key":"790_CR25","doi-asserted-by":"publisher","first-page":"8939","DOI":"10.1021\/acs.jpca.7b08750","volume":"121","author":"Jon Paul Janet","year":"2017","unstructured":"Janet Jon Paul, Kulik Heather J (2017) Resolving transition metal chemical space: feature selection for machine learning and structure-property relationships. J Phys Chem A 121(46):8939\u20138954. https:\/\/doi.org\/10.1021\/acs.jpca.7b08750","journal-title":"J Phys Chem A"},{"key":"790_CR26","unstructured":"Paszke A, Gross Sam, Massa Francisco, Lerer Adam, Bradbury James, Chanan Gregory, Killeen Trevor, Lin Zeming, Gimelshein Natalia, Antiga Luca, Desmaison Alban, Kopf Andreas, Yang Edward, DeVito Zachary, Raison Martin, Tejani Alykhan, Chilamkurthy Sasank, Steiner Benoit, Fang Lu, Bai Junjie, Chintala Soumith (2019) \u201cPyTorch: An Imperative Style, High-Performance Deep Learning Library\u201d. In: Advances in Neural Information Processing Systems 32. Curran Associates, Inc., pp. 8024\u20138035. https:\/\/dl.acm.org\/doi\/10.5555\/3454287.3455008"},{"key":"790_CR27","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"key":"790_CR28","doi-asserted-by":"crossref","unstructured":"Vazquez-Salazar Luis Itza, Boittier Eric D, Meuwly M. Uncertainty quantification for predictions of atomistic neural networks. 2022. arXiv: 2207.06916 [physics.chem-ph]","DOI":"10.1039\/D2SC04056E"},{"issue":"22","key":"790_CR29","doi-asserted-by":"publisher","first-page":"7866","DOI":"10.1039\/d0sc06805e","volume":"12","author":"David E Graff","year":"2021","unstructured":"Graff David E, Shakhnovich Eugene I, Coley Connor W (2021) Accelerating high-throughput virtual screening through molecular pool-based active learning. Chem Sci 12(22):7866\u20137881. https:\/\/doi.org\/10.1039\/d0sc06805e","journal-title":"Chem Sci"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-023-00790-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-023-00790-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-023-00790-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T10:03:43Z","timestamp":1702893823000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-023-00790-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,18]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["790"],"URL":"https:\/\/doi.org\/10.1186\/s13321-023-00790-0","relation":{"has-preprint":[{"id-type":"doi","id":"10.26434\/chemrxiv-2023-w93dm","asserted-by":"object"}]},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,18]]},"assertion":[{"value":"7 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"121"}}