{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T18:02:15Z","timestamp":1787162535060,"version":"build-2736575974"},"reference-count":32,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2020,8,18]],"date-time":"2020-08-18T00:00:00Z","timestamp":1597708800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,18]],"date-time":"2020-08-18T00:00:00Z","timestamp":1597708800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2020,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Quantum machine learning models have been gaining significant traction within atomistic simulation communities. Conventionally, relative model performances are being assessed and compared using learning curves (prediction error vs. training set size). This article illustrates the limitations of using the Mean Absolute Error (MAE) for benchmarking, which is particularly relevant in the case of non-normal error distributions. We analyze more specifically the prediction error distribution of the kernel ridge regression with SLATM representation and\n                    <jats:italic>L<\/jats:italic>\n                    <jats:sub>2<\/jats:sub>\n                    distance metric (KRR-SLATM-L2) for effective atomization energies of QM7b molecules calculated at the level of theory CCSD(T)\/cc-pVDZ. Error distributions of HF and MP2 at the same basis set referenced to CCSD(T) values were also assessed and compared to the KRR model. We show that the true performance of the KRR-SLATM-L2 method over the QM7b dataset is poorly assessed by the Mean Absolute Error, and can be notably improved after adaptation of the learning set.\n                  <\/jats:p>","DOI":"10.1088\/2632-2153\/aba184","type":"journal-article","created":{"date-parts":[[2020,6,30]],"date-time":"2020-06-30T18:39:18Z","timestamp":1593542358000},"page":"035011","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Impact of non-normal error distributions on the benchmarking and ranking of quantum machine learning models"],"prefix":"10.1088","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8586-6222","authenticated-orcid":false,"given":"Pascal","family":"Pernot","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7553-1645","authenticated-orcid":false,"given":"Bing","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8401-8037","authenticated-orcid":false,"given":"Andreas","family":"Savin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2020,8,18]]},"reference":[{"key":"mlstaba184bib1","doi-asserted-by":"publisher","first-page":"5288","DOI":"10.1021\/jp509980w","article-title":"Prediction uncertainty of density functional approximations for properties of crystals with cubic symmetry","volume":"119","author":"Pernot","year":"2015","journal-title":"J. Phys. Chem. A"},{"key":"mlstaba184bib2","doi-asserted-by":"publisher","DOI":"10.1063\/1.5016248","article-title":"Probabilistic performance estimators for computational chemistry methods: the empirical cumulative distribution function of absolute errors","volume":"148","author":"Pernot","year":"2018","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib3","doi-asserted-by":"publisher","first-page":"737","DOI":"10.1515\/zpch-2015-0713","article-title":"Intensive atomization energy: Re-thinking a metric for electronic structure theory methods","volume":"230","author":"Perdew","year":"2016","journal-title":"Z. Phys. Chem."},{"key":"mlstaba184bib4","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1016\/j.commatsci.2016.01.039","article-title":"Is the error on first-principles volume predictions absolute or relative?","volume":"117","author":"Lejaeghere","year":"2016","journal-title":"Comput. Mater. Sci."},{"key":"mlstaba184bib5","doi-asserted-by":"publisher","first-page":"5255","DOI":"10.1021\/acs.jctc.7b00577","article-title":"Prediction errors of molecular machine learning models lower than hybrid DFT error","volume":"15","author":"Faber","year":"2017","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaba184bib6","doi-asserted-by":"publisher","first-page":"1546","DOI":"10.1021\/acs.jctc.8b00832","article-title":"Boosting quantum machine learning models with a multilevel combination technique: Pople diagrams revisited","volume":"15","author":"Zaspel","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaba184bib7","doi-asserted-by":"publisher","DOI":"10.1063\/5.0006204","article-title":"Probabilistic performance estimators for computational chemistry methods: Systematic improvement probability and ranking probability matrix. II. Applications","volume":"152","author":"Pernot","year":"2020","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib8","first-page":"21","article-title":"Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests","volume":"2","author":"Mohd Razali","year":"2011","journal-title":"J. Stat. Model. Analytics"},{"key":"mlstaba184bib9","doi-asserted-by":"publisher","first-page":"204","DOI":"10.2478\/msr-2019-0026","article-title":"About not correcting for systematic effects","volume":"19","author":"Klauenberg","year":"2019","journal-title":"Meas. Sci. Rev."},{"key":"mlstaba184bib10","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1002\/qua.24605","article-title":"Uncertainty quantification in thermochemistry, benchmarking electronic structure computations and active thermochemical tables","volume":"114","author":"Ruscic","year":"2014","journal-title":"Int. J. Quantum Chem."},{"key":"mlstaba184bib11","doi-asserted-by":"publisher","DOI":"10.1063\/1.4932594","article-title":"How well do static electronic dipole polarizabilities from gas-phase experiments compare with density functional and MP2 computations?","volume":"143","author":"Thakkar","year":"2015","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib12","doi-asserted-by":"publisher","first-page":"16502","DOI":"10.1021\/jp960976r","article-title":"Harmonic vibrational frequencies: An evaluation of Hartree-Fock, M\u00f6ller-Plesset, quadratic configuration interaction, density functional theory and semiempirical scale factors","volume":"100","author":"Scott","year":"1996","journal-title":"J. Phys. Chem."},{"key":"mlstaba184bib13","doi-asserted-by":"publisher","DOI":"10.1063\/1.3086931","article-title":"Comment on \"Uncertainties in scaling factors for ab initio vibrational zero-point energies","volume":"130","author":"Pernot","year":"2009","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib13","doi-asserted-by":"publisher","DOI":"10.1063\/1.3581022","volume":"134","author":"Pernot","year":"2011","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib14","doi-asserted-by":"publisher","DOI":"10.1063\/1.3493630","article-title":"Calibration sets and the accuracy of vibrational scaling factorsA case study with the X3LYP hybrid functional","volume":"133","author":"Pernot","year":"2010","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib15","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevB.89.014304","article-title":"Ab initio based thermal property predictions at a low cost: An error analysis","volume":"89","author":"Lejaeghere","year":"2014","journal-title":"Phys. Rev. B"},{"key":"mlstaba184bib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/10408436.2013.772503","article-title":"Error estimates for solid-state density-functional theory predictions: An overview by means of the ground-state elemental crystals","volume":"39","author":"Lejaeghere","year":"2014","journal-title":"Crit. Rev. Solid State Mater. Sci."},{"key":"mlstaba184bib17","doi-asserted-by":"publisher","first-page":"3297","DOI":"10.1021\/acs.jctc.7b00235","article-title":"Reliable estimation of prediction uncertainty for physicochemical property models","volume":"13","author":"Proppe","year":"2017","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaba184bib18","doi-asserted-by":"publisher","first-page":"2087","DOI":"10.1021\/acs.jctc.5b00099","article-title":"Big data meets quantum chemistry approximations: The \u03b4-machine learning approach","volume":"11","author":"Ramakrishnan","year":"2015","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaba184bib19","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1557\/mrc.2019.107","article-title":"Machine learning prediction of accurate atomization energies of organic molecules from low-fidelity quantum chemical calculations","volume":"9","author":"Ward","year":"2019","journal-title":"MRS Commun."},{"key":"mlstaba184bib20","doi-asserted-by":"publisher","first-page":"6046","DOI":"10.1021\/acs.jctc.9b00627","article-title":"Gaussian process-based refinement of dispersion corrections","volume":"15","author":"Proppe","year":"2019","journal-title":"J. Chem. Theory Comput."},{"key":"mlstaba184bib21","author":"Core Team","year":"2019"},{"key":"mlstaba184bib22","author":"Canty","year":"2019"},{"key":"mlstaba184bib23","author":"Komsta","year":"2015"},{"key":"mlstaba184bib24","author":"Young","year":"2020"},{"key":"mlstaba184bib25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v032.i06","article-title":"mixtools: An R package for analyzing finite mixture models","volume":"32","author":"Benaglia","year":"2009","journal-title":"J. Stat. Softw"},{"key":"mlstaba184bib26","doi-asserted-by":"publisher","DOI":"10.1088\/1367-2630\/15\/9\/095003","article-title":"Machine learning of molecular electronic properties in chemical compound space","volume":"15","author":"Montavon","year":"2013","journal-title":"New J. Phys."},{"key":"mlstaba184bib27","doi-asserted-by":"publisher","DOI":"10.1063\/5.0006202","article-title":"Probabilistic performance estimators for computational chemistry methods: Systematic improvement probability and ranking probability matrix. I. Theory","volume":"152","author":"Pernot","year":"2020","journal-title":"J. Chem. Phys."},{"key":"mlstaba184bib28","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1021\/ed060p626","article-title":"Molecular formulas of organic compounds: the nitrogen rule and degree of unsaturation","volume":"60","author":"Pellegrin","year":"1983","journal-title":"J. Chem. Educ."},{"key":"mlstaba184bib29","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1021\/ci00057a005","article-title":"Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules","volume":"28","author":"Weininger","year":"1988","journal-title":"J. Chem. Inf. Comput. Sci."},{"key":"mlstaba184bib30","doi-asserted-by":"publisher","first-page":"2864","DOI":"10.1021\/ci300415d","article-title":"Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17","volume":"52","author":"Ruddigkeit","year":"2012","journal-title":"J. Chem. Inf. Model."},{"key":"mlstaba184bib31","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2014.22","article-title":"Quantum chemistry structures and properties of 134 kilo molecules","volume":"1","author":"Ramakrishnan","year":"2014","journal-title":"Scientific Data"}],"updated-by":[{"DOI":"10.1088\/2632-2153\/abc350","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000}}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184","content-type":"text\/html","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T19:02:58Z","timestamp":1645038178000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/aba184"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,18]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,8,18]]},"published-print":{"date-parts":[[2020,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/aba184","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,18]]},"assertion":[{"value":"Impact of non-normal error distributions on the benchmarking and ranking of quantum machine learning models","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2020 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-04-02","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-06-30","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-08-18","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}