{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T05:56:03Z","timestamp":1740894963050,"version":"3.38.0"},"reference-count":42,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2024,11,18]],"date-time":"2024-11-18T00:00:00Z","timestamp":1731888000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"CAMPA, ComPASS, and NUCLEI SciDAC projects within DOE\u2019s Office of Science, Advanced Scientific Computing Research","award":["DE-AC02-05CH11231"],"award-info":[{"award-number":["DE-AC02-05CH11231"]}]},{"name":"CAMPA, ComPASS, and NUCLEI SciDAC projects within DOE\u2019s Office of Science, Advanced Scientific Computing Research","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}]},{"name":"PETSc\/TAO activity within Exascale Computing Project","award":["17-SC-20-SC"],"award-info":[{"award-number":["17-SC-20-SC"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:p> libEnsemble is a Python-based toolkit for running dynamic ensembles, developed as part of the DOE Exascale Computing Project. The toolkit utilizes a unique generator\u2013simulator\u2013allocator paradigm, where generators produce input for simulators, simulators evaluate those inputs, and allocators decide whether and when a simulator or generator should be called. The generator steers the ensemble based on simulation results. Generators may, for example, apply methods for numerical optimization, machine learning, or statistical calibration. libEnsemble communicates between a manager and workers. Flexibility is provided through multiple manager\u2013worker communication substrates each of which has different benefits. These include Python\u2019s multiprocessing, mpi4py, and TCP. Multisite ensembles are supported using Balsam or Globus Compute. We overview the unique characteristics of libEnsemble as well as current and potential interoperability with other packages in the workflow ecosystem. We highlight libEnsemble\u2019s dynamic resource features: libEnsemble can detect system resources, such as available nodes, cores, and GPUs, and assign these in a portable way. These features allow users to specify the number of processors and GPUs required for each simulation; and resources will be automatically assigned on a wide range of systems, including Frontier, Aurora, and Perlmutter. Such ensembles can include multiple simulation types, some using GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also describe the benefits of libEnsemble\u2019s generator\u2013simulator coupling, which easily exposes to the user the ability to cancel, and portably kill, running simulations based on models that are updated with intermediate simulation output. We demonstrate libEnsemble\u2019s capabilities, scalability, and scientific impact via a Gaussian process surrogate training problem for the longitudinal density profile at the exit of a plasma accelerator stage. The study uses gpCAM for the surrogate model and employs either Wake-T or WarpX simulations, highlighting efficient use of resources that can easily extend to exascale. <\/jats:p>","DOI":"10.1177\/10943420241298297","type":"journal-article","created":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T05:44:26Z","timestamp":1731995066000},"page":"177-192","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Portable, heterogeneous ensemble workflows at scale using libEnsemble"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7500-6138","authenticated-orcid":false,"given":"Stephen","family":"Hudson","sequence":"first","affiliation":[{"name":"MCS, Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9924-2082","authenticated-orcid":false,"given":"Jeffrey","family":"Larson","sequence":"additional","affiliation":[{"name":"MCS, Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9916-9038","authenticated-orcid":false,"given":"John-Luke","family":"Navarro","sequence":"additional","affiliation":[{"name":"MCS, Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6099-2772","authenticated-orcid":false,"given":"Stefan M","family":"Wild","sequence":"additional","affiliation":[{"name":"AMCR, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"},{"name":"IEMS, Northwestern University, Evanston, IL, USA"}]}],"member":"179","published-online":{"date-parts":[[2024,11,18]]},"reference":[{"doi-asserted-by":"publisher","key":"bibr1-10943420241298297","DOI":"10.1145\/3307681.3325400"},{"doi-asserted-by":"publisher","key":"bibr2-10943420241298297","DOI":"10.1109\/icpp.2016.59"},{"doi-asserted-by":"publisher","key":"bibr3-10943420241298297","DOI":"10.2172\/2205494"},{"doi-asserted-by":"publisher","key":"bibr4-10943420241298297","DOI":"10.14529\/jsfi170104"},{"volume-title":"High-Dimensional Gaussian Process Methods for Uncertainty Quantification","year":"2023","author":"Chan MY","key":"bibr5-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr6-10943420241298297","DOI":"10.1080\/00401706.2023.2210170"},{"key":"bibr7-10943420241298297","first-page":"06881","volume":"2304","author":"Chang TH","year":"2023","journal-title":"Technical Report"},{"doi-asserted-by":"publisher","key":"bibr8-10943420241298297","DOI":"10.21105\/joss.04468"},{"doi-asserted-by":"publisher","key":"bibr9-10943420241298297","DOI":"10.22360\/springsim.2020.hpc.001"},{"doi-asserted-by":"publisher","key":"bibr10-10943420241298297","DOI":"10.1145\/3369583.3392683"},{"unstructured":"Colvin S (2017\u20132024) Pydantic: data validation and settings management using Python type hinting. Available at: https:\/\/github.com\/samuelcolvin\/pydantic.","key":"bibr11-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr12-10943420241298297","DOI":"10.5281\/zenodo.5903364"},{"doi-asserted-by":"publisher","key":"bibr13-10943420241298297","DOI":"10.1109\/MCSE.2021.3083216"},{"doi-asserted-by":"publisher","key":"bibr14-10943420241298297","DOI":"10.1109\/SC41404.2022.00008"},{"doi-asserted-by":"publisher","key":"bibr15-10943420241298297","DOI":"10.18429\/JACoW-IPAC2022-WEPOST030"},{"doi-asserted-by":"publisher","key":"bibr16-10943420241298297","DOI":"10.1088\/1742-6596\/1350\/1\/012056"},{"doi-asserted-by":"publisher","key":"bibr17-10943420241298297","DOI":"10.1103\/PhysRevAccelBeams.26.084601"},{"doi-asserted-by":"publisher","key":"bibr18-10943420241298297","DOI":"10.1201\/9780367815493"},{"doi-asserted-by":"publisher","key":"bibr19-10943420241298297","DOI":"10.1038\/s41586-020-2649-2"},{"doi-asserted-by":"publisher","key":"bibr20-10943420241298297","DOI":"10.1109\/e-Science58273.2023.10254912"},{"doi-asserted-by":"publisher","key":"bibr21-10943420241298297","DOI":"10.1109\/tpds.2021.3082815"},{"unstructured":"Hudson S, Larson J, Wild SM, et al. (2022b) libEnsemble users manual. Available at: https:\/\/buildmedia.readthedocs.org\/media\/pdf\/libensemble\/latest\/libensemble.pdf.","key":"bibr22-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr23-10943420241298297","DOI":"10.21105\/joss.06031"},{"unstructured":"Hudson S, Larson J, Wild SM, et al. (2024) libEnsemble. Available at: https:\/\/github.com\/Libensemble\/libEnsemble.","key":"bibr24-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr25-10943420241298297","DOI":"10.1007\/s12532-017-0131-4"},{"unstructured":"libEnsemble Community (2023) A selection of libEnsemble functions and complete workflows from the community. Available at: https:\/\/github.com\/Libensemble\/libe-community-examples.","key":"bibr26-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr27-10943420241298297","DOI":"10.1145\/3307681.3326610"},{"doi-asserted-by":"publisher","key":"bibr28-10943420241298297","DOI":"10.48550\/arXiv.1712.05889"},{"doi-asserted-by":"publisher","key":"bibr29-10943420241298297","DOI":"10.18429\/JACoW-ICAP2018-SAPAF03"},{"doi-asserted-by":"publisher","key":"bibr30-10943420241298297","DOI":"10.1016\/j.cpc.2022.108566"},{"doi-asserted-by":"publisher","key":"bibr31-10943420241298297","DOI":"10.1201\/9781003359593"},{"doi-asserted-by":"publisher","key":"bibr32-10943420241298297","DOI":"10.5281\/zenodo.10393189"},{"doi-asserted-by":"publisher","key":"bibr33-10943420241298297","DOI":"10.1038\/s41598-023-30062-8"},{"doi-asserted-by":"publisher","key":"bibr34-10943420241298297","DOI":"10.1145\/3581784.3607047"},{"unstructured":"Plumlee M, S\u00fcrer O, Wild SM, et al. (2023) Surmise 0.2.1 users manual. Available at: https:\/\/surmise.readthedocs.io.","key":"bibr35-10943420241298297"},{"unstructured":"RadiaSoft (2024) rsopt. Available at: https:\/\/github.com\/radiasoft\/rsopt.","key":"bibr36-10943420241298297"},{"doi-asserted-by":"publisher","key":"bibr37-10943420241298297","DOI":"10.25080\/majora-7b98e3ed-013"},{"doi-asserted-by":"publisher","key":"bibr38-10943420241298297","DOI":"10.48550\/arXiv.1909.08704"},{"doi-asserted-by":"publisher","key":"bibr39-10943420241298297","DOI":"10.1080\/00401706.2023.2246157"},{"doi-asserted-by":"publisher","key":"bibr40-10943420241298297","DOI":"10.1038\/ncomms7860"},{"doi-asserted-by":"publisher","key":"bibr41-10943420241298297","DOI":"10.1109\/MLHPC54614.2021.00007"},{"doi-asserted-by":"publisher","key":"bibr42-10943420241298297","DOI":"10.1177\/10943420241286476"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241298297","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420241298297","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241298297","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T20:39:34Z","timestamp":1740861574000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420241298297"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,18]]},"references-count":42,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10.1177\/10943420241298297"],"URL":"https:\/\/doi.org\/10.1177\/10943420241298297","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"type":"print","value":"1094-3420"},{"type":"electronic","value":"1741-2846"}],"subject":[],"published":{"date-parts":[[2024,11,18]]}}}