{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T22:40:39Z","timestamp":1783118439037,"version":"3.54.6"},"reference-count":92,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2021,7,12]],"date-time":"2021-07-12T00:00:00Z","timestamp":1626048000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,12]],"date-time":"2021-07-12T00:00:00Z","timestamp":1626048000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"Jacques-Emile Dubois Student Dissertation Fellowship"},{"name":"Herchel Smith Graduate Fellowship"},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","award":["PGSD3-534584-2019"],"award-info":[{"award-number":["PGSD3-534584-2019"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Vector Institute for Artificial Intelligence"},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"crossref","award":["HR00111920027"],"award-info":[{"award-number":["HR00111920027"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"crossref","award":["N00014-19-1-2134"],"award-info":[{"award-number":["N00014-19-1-2134"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is <jats:italic>a priori<\/jats:italic> unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce <jats:sc>Olympus<\/jats:sc>, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. <jats:sc>Olympus<\/jats:sc> includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly Python interface. Furthermore, <jats:sc>Olympus<\/jats:sc> facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, <jats:sc>Olympus<\/jats:sc> mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies.<\/jats:p>","DOI":"10.1088\/2632-2153\/abedc8","type":"journal-article","created":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T22:15:32Z","timestamp":1615500932000},"page":"035021","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":51,"title":["Olympus: a benchmarking framework for noisy optimization and experiment planning"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3711-9761","authenticated-orcid":false,"given":"Florian","family":"H\u00e4se","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0019-8806","authenticated-orcid":false,"given":"Matteo","family":"Aldeghi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5762-1006","authenticated-orcid":false,"given":"Riley J","family":"Hickman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1771-2023","authenticated-orcid":false,"given":"Lo\u00efc M","family":"Roch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melodie","family":"Christensen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Liles","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4345-3005","authenticated-orcid":false,"given":"Jason E","family":"Hein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8277-4434","authenticated-orcid":false,"given":"Al\u00e1n","family":"Aspuru-Guzik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2021,7,12]]},"reference":[{"key":"mlstabedc8bib1","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/4235.585893","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"mlstabedc8bib2","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/S0304-3975(02)00094-4","volume":"287","author":"Droste","year":"2002","journal-title":"Theor. Comput. Sci."},{"key":"mlstabedc8bib3","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1021\/acs.accounts.0c00674","volume":"54","author":"Wilbraham","year":"2021","journal-title":"Acc. Chem. Res."},{"key":"mlstabedc8bib4","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1021\/acsenergylett.8b02278","volume":"4","author":"Krishnamurthy","year":"2018","journal-title":"ACS Energy Lett."},{"key":"mlstabedc8bib5","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1021\/acscentsci.7b00550","volume":"4","author":"Aspuru-Guzik","year":"2018","journal-title":"ACS Cent. Sci."},{"key":"mlstabedc8bib6","doi-asserted-by":"publisher","first-page":"6107","DOI":"10.1021\/acs.chemrev.5b00691","volume":"116","author":"Le","year":"2016","journal-title":"Chem. Rev."},{"key":"mlstabedc8bib7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.coche.2015.07.001","volume":"9","author":"Houben","year":"2015","journal-title":"Curr. Opin. Chem. Eng."},{"key":"mlstabedc8bib8","doi-asserted-by":"publisher","first-page":"944","DOI":"10.1038\/40099","volume":"389","author":"Danielson","year":"1997","journal-title":"Nature"},{"key":"mlstabedc8bib9","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1038\/s41578-018-0005-z","volume":"3","author":"Tabor","year":"2018","journal-title":"Nat. Rev. Mater."},{"key":"mlstabedc8bib10","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1016\/j.trechm.2019.02.007","volume":"1","author":"H\u00e4se","year":"2019","journal-title":"Trends Chem."},{"key":"mlstabedc8bib11","doi-asserted-by":"publisher","first-page":"9640","DOI":"10.1039\/C9SC03766G","volume":"10","author":"Stein","year":"2019","journal-title":"Chem. Sci."},{"key":"mlstabedc8bib12","doi-asserted-by":"publisher","DOI":"10.1002\/anie.201909987","volume":"59","author":"Coley","year":"2020","journal-title":"Angew. Chem., Int. Ed."},{"key":"mlstabedc8bib13","doi-asserted-by":"publisher","DOI":"10.1002\/anie.201909989","volume":"59","author":"Coley","year":"2020","journal-title":"Angew. Chem. Int. Ed."},{"key":"mlstabedc8bib14","doi-asserted-by":"publisher","DOI":"10.1002\/adma.201907801","volume":"32","author":"Langner","year":"2020","journal-title":"Adv. Mater."},{"key":"mlstabedc8bib15","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1038\/s41586-018-0307-8","volume":"559","author":"Granda","year":"2018","journal-title":"Nature"},{"key":"mlstabedc8bib16","doi-asserted-by":"publisher","first-page":"eaaz8867","DOI":"10.1126\/sciadv.aaz8867","volume":"6","author":"MacLeod","year":"2020","journal-title":"Sci. Adv."},{"key":"mlstabedc8bib17","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1038\/s41586-020-2442-2","volume":"583","author":"Burger","year":"2020","journal-title":"Nature"},{"key":"mlstabedc8bib18","doi-asserted-by":"publisher","first-page":"eaaz1708","DOI":"10.1126\/sciadv.aaz1708","volume":"6","author":"Gongora","year":"2020","journal-title":"Sci. Adv."},{"key":"mlstabedc8bib19","doi-asserted-by":"publisher","first-page":"658","DOI":"10.1039\/C6RE00153J","volume":"1","author":"Reizman","year":"2016","journal-title":"React. Chem. Eng."},{"key":"mlstabedc8bib20","doi-asserted-by":"publisher","first-page":"641","DOI":"10.3762\/bjoc.10.56","volume":"10","author":"Ingham","year":"2014","journal-title":"Beilstein J. Org. Chem."},{"key":"mlstabedc8bib21","doi-asserted-by":"publisher","first-page":"1434","DOI":"10.1039\/b711412e","volume":"7","author":"Krishnadasan","year":"2007","journal-title":"Lab Chip"},{"key":"mlstabedc8bib22","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1039\/C7RE00123A","volume":"2","author":"Walker","year":"2017","journal-title":"React. Chem. Eng."},{"key":"mlstabedc8bib23","doi-asserted-by":"publisher","first-page":"1220","DOI":"10.1126\/science.aat0650","volume":"361","author":"B\u00e9dard","year":"2018","journal-title":"Science"},{"key":"mlstabedc8bib24","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1021\/acs.oprd.5b00313","volume":"20","author":"Fitzpatrick","year":"2016","journal-title":"Org. Process Res. Dev."},{"key":"mlstabedc8bib25","doi-asserted-by":"publisher","DOI":"10.1021\/acs.joc.8b01821","volume":"83","author":"Cort\u00e9s-Borda","year":"2018","journal-title":"J. Org. Chem."},{"key":"mlstabedc8bib26","doi-asserted-by":"publisher","first-page":"1169","DOI":"10.1021\/op100123e","volume":"14","author":"McMullen","year":"2010","journal-title":"Org. Process Res. Dev."},{"key":"mlstabedc8bib27","doi-asserted-by":"publisher","DOI":"10.1038\/ncomms11241","volume":"7","author":"Xue","year":"2016","journal-title":"Nat. Commun."},{"key":"mlstabedc8bib28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-48114-3","volume":"9","author":"Noack","year":"2019","journal-title":"Sci. Rep."},{"key":"mlstabedc8bib29","doi-asserted-by":"publisher","DOI":"10.1038\/srep25890","volume":"6","author":"Wigley","year":"2016","journal-title":"Sci. Rep."},{"key":"mlstabedc8bib30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/npjcompumats.2016.31","volume":"2","author":"Nikolaev","year":"2016","journal-title":"npj Comput. Mater."},{"key":"mlstabedc8bib31","doi-asserted-by":"publisher","first-page":"2696","DOI":"10.1039\/C9SC05999G","volume":"11","author":"Rohr","year":"2020","journal-title":"Chem. Sci."},{"key":"mlstabedc8bib32","doi-asserted-by":"publisher","first-page":"7642","DOI":"10.1039\/C8SC02239A","volume":"9","author":"H\u00e4se","year":"2018","journal-title":"Chem. Sci."},{"key":"mlstabedc8bib33","author":"Elhara","year":"2019"},{"key":"mlstabedc8bib34","author":"Brockman","year":"2016"},{"key":"mlstabedc8bib35","author":"Hertel","year":"2020"},{"key":"mlstabedc8bib36","first-page":"pp 2623","author":"Akiba","year":"2019"},{"key":"mlstabedc8bib37","doi-asserted-by":"publisher","first-page":"2338","DOI":"10.21105\/joss.02338","volume":"5","author":"Biscani","year":"2020","journal-title":"J. Open Source Softw."},{"key":"mlstabedc8bib38","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1002\/cmtd.20","volume":"1","author":"Felton","year":"2021","journal-title":"Chemistry\u2014Methods"},{"key":"mlstabedc8bib39","volume":"vol 2","author":"LeCun","year":"2010"},{"key":"mlstabedc8bib40","author":"Krizhevsky","year":"2009"},{"key":"mlstabedc8bib41","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1039\/C7SC02664A","volume":"9","author":"Wu","year":"2018","journal-title":"Chem. Sci."},{"key":"mlstabedc8bib42","doi-asserted-by":"publisher","first-page":"8732","DOI":"10.1021\/ja902302h","volume":"131","author":"Blum","year":"2009","journal-title":"J. Am. Chem. Soc."},{"key":"mlstabedc8bib43","doi-asserted-by":"publisher","first-page":"2864","DOI":"10.1021\/ci300415d","volume":"52","author":"Ruddigkeit","year":"2012","journal-title":"J. Chem. Inf. Model."},{"key":"mlstabedc8bib44","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.108.058301","volume":"108","author":"Rupp","year":"2012","journal-title":"Phys. Rev. Lett."},{"key":"mlstabedc8bib45","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2014.22","volume":"1","author":"Ramakrishnan","year":"2014","journal-title":"Sci. Data"},{"key":"mlstabedc8bib46","doi-asserted-by":"publisher","DOI":"10.1063\/1.4928757","volume":"143","author":"Ramakrishnan","year":"2015","journal-title":"J. Chem. Phys."},{"key":"mlstabedc8bib47","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/s13321-019-0391-2","volume":"11","author":"Glavatskikh","year":"2019","journal-title":"J. Cheminformatics"},{"key":"mlstabedc8bib48","doi-asserted-by":"publisher","first-page":"1096","DOI":"10.1021\/acs.jcim.8b00839","volume":"59","author":"Brown","year":"2019","journal-title":"J. Chem. Inf. Model."},{"key":"mlstabedc8bib49","author":"Polykovskiy","year":"2018"},{"key":"mlstabedc8bib50","article-title":"Gpyopt: a Bayesian optimization framework in python","author":"","year":"2016"},{"key":"mlstabedc8bib51","first-page":"pp 115","author":"Bergstra","year":"2013"},{"key":"mlstabedc8bib52","doi-asserted-by":"publisher","first-page":"281","DOI":"10.5555\/2188385.2188395","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"mlstabedc8bib53","first-page":"pp 2546","author":"Bergstra","year":"2011"},{"key":"mlstabedc8bib54","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1021\/acscentsci.8b00307","volume":"4","author":"H\u00e4se","year":"2018","journal-title":"ACS Cent. Sci."},{"key":"mlstabedc8bib55","first-page":"2171","volume":"13","author":"Fortin","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"mlstabedc8bib56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/106365603321828970","volume":"11","author":"Hansen","year":"2003","journal-title":"Evol. Comput."},{"key":"mlstabedc8bib57","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1162\/106365601750190398","volume":"9","author":"Hansen","year":"2001","journal-title":"Evol. Comput."},{"key":"mlstabedc8bib58","first-page":"pp 39","author":"Eberhart","year":"1995"},{"key":"mlstabedc8bib59","first-page":"pp 69","author":"Shi","year":"1998"},{"key":"mlstabedc8bib60","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1023\/A:1008202821328","volume":"11","author":"Storn","year":"1997","journal-title":"J. Glob. Optim."},{"key":"mlstabedc8bib61","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1090\/qam\/10667","volume":"2","author":"Curry","year":"1944","journal-title":"Q. Appl. Math."},{"key":"mlstabedc8bib62","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1016\/j.procs.2015.05.241","volume":"51","author":"Bouwmeester","year":"2015","journal-title":"Proc. Comput. Sci."},{"key":"mlstabedc8bib63","doi-asserted-by":"publisher","first-page":"409","DOI":"10.6028\/jres.049.044","volume":"49","author":"Hestenes","year":"1952","journal-title":"J. Res. Natl Bur. Stand."},{"key":"mlstabedc8bib64","doi-asserted-by":"publisher","first-page":"550","DOI":"10.1145\/279232.279236","volume":"23","author":"Zhu","year":"1997","journal-title":"ACM Trans. Math. Softw."},{"key":"mlstabedc8bib65","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1137\/0916069","volume":"16","author":"Byrd","year":"1995","journal-title":"SIAM J. Sci. Comput."},{"key":"mlstabedc8bib66","first-page":"pp 529","author":"Nocedal","year":"2006"},{"key":"mlstabedc8bib67","article-title":"A software package for sequential quadratic programming","author":"Kraft","year":"1988"},{"key":"mlstabedc8bib68","author":"Anderson","year":"2016"},{"key":"mlstabedc8bib69","volume":"vol 2","author":"Box","year":"2005"},{"key":"mlstabedc8bib70","author":"Fisher","year":"1937"},{"key":"mlstabedc8bib71","first-page":"784","volume":"7","author":"Sobol\u2019","year":"1967","journal-title":"Zh. Vychisl. Mat. Mat. Fiz."},{"key":"mlstabedc8bib72","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1377612.1377613","volume":"35","author":"Huyer","year":"2008","journal-title":"ACM Trans. Math. Softw."},{"key":"mlstabedc8bib73","doi-asserted-by":"publisher","first-page":"5111","DOI":"10.1021\/jp970984n","volume":"101","author":"Wales","year":"1997","journal-title":"J. Phys. Chem. A"},{"key":"mlstabedc8bib74","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1093\/comjnl\/7.4.308","volume":"7","author":"Nelder","year":"1965","journal-title":"Comput. J."},{"key":"mlstabedc8bib75","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1561\/2200000050","volume":"8","author":"Bubeck","year":"2015","journal-title":"Found. Trends Mach. Learn."},{"key":"mlstabedc8bib76","author":"Boyd","year":"2004"},{"key":"mlstabedc8bib77","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/0098-1354(90)87072-W","volume":"14","author":"Lucia","year":"1990","journal-title":"Comput. Chem. Eng."},{"key":"mlstabedc8bib78","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1007\/BF00935752","volume":"33","author":"Baba","year":"1981","journal-title":"J. Optim. Theory Appl."},{"key":"mlstabedc8bib79","first-page":"246","volume":"26","author":"Matyas","year":"1965","journal-title":"Autom. Remote Control"},{"key":"mlstabedc8bib80","first-page":"pp 83","author":"Rechenberg","year":"1978"},{"key":"mlstabedc8bib81","first-page":"pp 123","author":"Schwefel","year":"1977"},{"key":"mlstabedc8bib82","first-page":"301","volume":"3","author":"Zames","year":"1981","journal-title":"Inf. Technol. J."},{"key":"mlstabedc8bib83","volume":"vol 1","author":"Koza","year":"1992"},{"key":"mlstabedc8bib84","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1109\/2.294849","volume":"27","author":"Srinivas","year":"1994","journal-title":"Computer"},{"key":"mlstabedc8bib85","first-page":"949","volume":"15","author":"Wierstra","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"mlstabedc8bib86","volume":"vol 37","author":"Mockus","year":"2012"},{"key":"mlstabedc8bib87","first-page":"p 2","volume":"vol 2","author":"Mockus","year":"1978"},{"key":"mlstabedc8bib88","first-page":"pp 400","author":"Mo\u010dkus","year":"1975"},{"key":"mlstabedc8bib89","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0229862","volume":"15","author":"Roch","year":"2020","journal-title":"PLoS One"},{"key":"mlstabedc8bib90","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1039\/C7RE00123A","volume":"2","author":"Walker","year":"2017","journal-title":"React. Chem. Eng."},{"key":"mlstabedc8bib91","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/j.cej.2018.07.031","volume":"352","author":"Schweidtmann","year":"2018","journal-title":"Chem. Eng. J."},{"key":"mlstabedc8bib92","article-title":"Olympus: a benchmarking framework for noisy optimization and experiment planning","author":"","year":"2020"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T17:15:02Z","timestamp":1639415702000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/abedc8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,12]]},"references-count":92,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,7,12]]},"published-print":{"date-parts":[[2021,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/abedc8","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,12]]},"assertion":[{"value":"Olympus: a benchmarking framework for noisy optimization and experiment planning","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2020-10-23","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-03-11","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-07-12","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}