{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T07:44:37Z","timestamp":1768031077403,"version":"3.49.0"},"reference-count":43,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T00:00:00Z","timestamp":1768694400000},"content-version":"vor","delay-in-days":365,"URL":"http:\/\/www.sagepub.com\/licence-information-for-chorus"}],"funder":[{"DOI":"10.13039\/100000083","name":"Directorate for Computer and Information Science and Engineering","doi-asserted-by":"publisher","award":["1931524"],"award-info":[{"award-number":["1931524"]}],"id":[{"id":"10.13039\/100000083","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006132","name":"Office of Science","doi-asserted-by":"publisher","award":["17-SC-20-SC"],"award-info":[{"award-number":["17-SC-20-SC"]}],"id":[{"id":"10.13039\/100006132","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:p> The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization via the Toolkit for Advanced Optimization (TAO). PETSc is used in dozens of scientific fields and is an important building block for many simulation codes. During the U.S. Department of Energy\u2019s Exascale Computing Project, the PETSc team has made substantial efforts to enable efficient utilization of the massive fine-grain parallelism present within exascale compute nodes and to enable performance portability across exascale architectures. We recap some of the challenges that designers of numerical libraries face in such an endeavor, and then discuss the many developments we have made, which include the addition of new GPU backends, features supporting efficient on-device matrix assembly, better support for asynchronicity and GPU kernel concurrency, and new communication infrastructure. We evaluate the performance of these developments on some pre-exascale systems as well as the early exascale systems Frontier and Aurora, using compute kernel, communication layer, solver, and mini-application benchmark studies, and then close with a few observations drawn from our experiences on the tension between portable performance and other goals of numerical libraries. <\/jats:p>","DOI":"10.1177\/10943420241303710","type":"journal-article","created":{"date-parts":[[2025,1,18]],"date-time":"2025-01-18T12:24:07Z","timestamp":1737203047000},"page":"306-325","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":3,"title":["PETSc\/TAO developments for GPU-based early exascale systems"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0683-6899","authenticated-orcid":false,"given":"Richard Tran","family":"Mills","sequence":"first","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"given":"Mark F.","family":"Adams","sequence":"additional","affiliation":[{"name":"Lawrence Berkeley National Laboratory, Berkeley, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4407-205X","authenticated-orcid":false,"given":"Satish","family":"Balay","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9945-0639","authenticated-orcid":false,"given":"Jed","family":"Brown","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}]},{"given":"Jacob","family":"Faibussowitsch","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"given":"Toby","family":"Isaac","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2292-0735","authenticated-orcid":false,"given":"Matthew G.","family":"Knepley","sequence":"additional","affiliation":[{"name":"University at Buffalo, Buffalo, NY, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0030-3648","authenticated-orcid":false,"given":"Todd","family":"Munson","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"given":"Hansol","family":"Suh","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0435-0433","authenticated-orcid":false,"given":"Stefano","family":"Zampini","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology, Thuwal, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7720-0049","authenticated-orcid":false,"given":"Hong","family":"Zhang","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0367-2358","authenticated-orcid":false,"given":"Junchao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]}],"member":"179","published-online":{"date-parts":[[2025,1,18]]},"reference":[{"key":"bibr1-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/17M1118828"},{"key":"bibr2-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2022.3169974"},{"key":"bibr3-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS53621.2022.00020"},{"key":"bibr4-10943420241303710","volume-title":"A Performance Portable, Fully Implicit Landau Collision Operator with Batched Linear Solvers","author":"Adams MF","year":"2024"},{"key":"bibr5-10943420241303710","unstructured":"AMD (2024) HIP programming manual. URL. https:\/\/rocm.docs.amd.com\/projects\/HIP\/en\/latest\/user_guide\/programming_manual.html."},{"key":"bibr6-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/S0895479899358194"},{"key":"bibr7-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1145\/3480935"},{"key":"bibr8-10943420241303710","unstructured":"Balay S, Abhyankar S, Adams MF, et al. (2024) PETSc Users Manual. Technical Report ANL-21\/39 - Revision 3.21, Argonne National Laboratory. URL. https:\/\/petsc.org."},{"key":"bibr9-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/P3HPC49587.2019.00012"},{"key":"bibr10-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW50202.2020.00067"},{"key":"bibr11-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/ISPDC.2012.16"},{"key":"bibr12-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1007\/BF01582063"},{"key":"bibr13-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1002\/nme.2989"},{"key":"bibr14-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/0801001"},{"key":"bibr15-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1145\/2049662.2049670"},{"key":"bibr16-10943420241303710","unstructured":"Demmel J, Gilbert J, Li X (2024) SuperLU github. URL. https:\/\/github.com\/xiaoyeli\/superlu."},{"key":"bibr17-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22744-9_39"},{"key":"bibr18-10943420241303710","unstructured":"E4S Team (2024) E4S Web page. https:\/\/e4s.io."},{"key":"bibr19-10943420241303710","author":"Faibussowitsch J","year":"2023","journal-title":"arXiv preprint arXiv:2306.17801"},{"key":"bibr20-10943420241303710","unstructured":"Falgout R (2023) Hypre Users Manual. Technical Report Revision 2.28, Lawrence Livermore National Laboratory. URL. https:\/\/hypre.readthedocs.io\/."},{"key":"bibr21-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/0914029"},{"key":"bibr22-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2016.03.064"},{"key":"bibr23-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1063\/1.4979122"},{"key":"bibr24-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1016\/j.camwa.2021.01.003"},{"key":"bibr25-10943420241303710","unstructured":"Khronos SYCL Working Group (2020) SYCL 2020 specification. URL. https:\/\/registry.khronos.org\/SYCL\/specs\/sycl-2020\/pdf\/sycl-2020.pdf."},{"key":"bibr26-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2018.2875366"},{"key":"bibr27-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2023.3249110"},{"key":"bibr28-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1007\/BF01589116"},{"key":"bibr29-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2021.102831"},{"key":"bibr30-10943420241303710","unstructured":"NVIDA (2024) CUDA C++ programming guide. URL. https:\/\/docs.nvidia.com\/cuda\/pdf\/CUDA_C_Programming_Guide.pdf."},{"key":"bibr31-10943420241303710","unstructured":"NVIDIA (2024) NVIDIA OpenSHMEM library (NVSHMEM) documentation. URL. https:\/\/docs.nvidia.com\/nvshmem\/api\/index.html."},{"key":"bibr32-10943420241303710","unstructured":"Open Source Software Solutions, Inc (2020) OpenSHMEM application programming interface v1.5. URL. https:\/\/www.openshmem.org\/."},{"key":"bibr33-10943420241303710","unstructured":"OpenMP Architecture Review Board (2021) OpenMP application programming interface. URL. https:\/\/www.openmp.org\/wp-content\/uploads\/OpenMP-API-Specification-5-2.pdf."},{"key":"bibr34-10943420241303710","unstructured":"Panda D (2024) OSU microbenchmarks v7.3. https:\/\/mvapich.cse.ohio-state.edu\/benchmarks\/."},{"key":"bibr35-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/15M1026419"},{"key":"bibr36-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3097283"},{"key":"bibr37-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2023.103051"},{"key":"bibr38-10943420241303710","unstructured":"xSDK Team (2024) xSDK Web page. https:\/\/xsdk.info."},{"key":"bibr39-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1145\/3659914.3659931"},{"key":"bibr40-10943420241303710","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01370"},{"key":"bibr41-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3084070"},{"key":"bibr42-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1137\/21M140078X"},{"key":"bibr43-10943420241303710","doi-asserted-by":"publisher","DOI":"10.1145\/3555819.3555820"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241303710","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420241303710","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241303710","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241303710","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T18:13:29Z","timestamp":1741112009000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420241303710"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,18]]},"references-count":43,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["10.1177\/10943420241303710"],"URL":"https:\/\/doi.org\/10.1177\/10943420241303710","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,18]]}}}