{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T13:58:09Z","timestamp":1762869489407,"version":"3.38.0"},"reference-count":33,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T00:00:00Z","timestamp":1767916800000},"content-version":"vor","delay-in-days":365,"URL":"http:\/\/www.sagepub.com\/licence-information-for-chorus"}],"funder":[{"name":"Exascale Computing Project","award":["17-SC-20-SC"],"award-info":[{"award-number":["17-SC-20-SC"]}]},{"name":"U.S. Department of Energy, Office of Science","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}]},{"name":"U.S. DOE Office of Science Advanced Scientific Computing Research Program","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}]},{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"crossref","award":["DE-AC05-00OR22725"],"award-info":[{"award-number":["DE-AC05-00OR22725"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:p> The advent of exascale supercomputers heralds a new era of scientific discovery, yet it introduces significant architectural challenges that must be overcome for MPI applications to fully exploit its potential. Among these challenges is the adoption of heterogeneous architectures, particularly the integration of GPUs to accelerate computation. Additionally, the complexity of multithreaded programming models has also become a critical factor in achieving performance at scale. The efficient utilization of hardware acceleration for communication, provided by modern NICs, is also essential for achieving low latency and high throughput communication in such complex systems. In response to these challenges, the MPICH library, a high-performance and widely used Message Passing Interface (MPI) implementation, has undergone significant enhancements. This paper presents four major contributions that prepare MPICH for the exascale transition. First, we describe a lightweight communication stack that leverages the advanced features of modern NICs to maximize hardware acceleration. Second, our work showcases a highly scalable multithreaded communication model that addresses the complexities of concurrent environments. Third, we introduce GPU-aware communication capabilities that optimize data movement in GPU-integrated systems. Finally, we present a new datatype engine aimed at accelerating the use of MPI derived datatypes on GPUs. These improvements in the MPICH library not only address the immediate needs of exascale computing architectures but also set a foundation for exploiting future innovations in high-performance computing. By embracing these new designs and approaches, MPICH-derived libraries from HPE Cray and Intel were able to achieve real exascale performance on OLCF Frontier and ALCF Aurora respectively. <\/jats:p>","DOI":"10.1177\/10943420241311608","type":"journal-article","created":{"date-parts":[[2025,1,9]],"date-time":"2025-01-09T18:46:07Z","timestamp":1736448367000},"page":"283-305","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":3,"title":["Preparing MPICH for exascale"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3731-5423","authenticated-orcid":false,"given":"Yanfei","family":"Guo","sequence":"first","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4705-2713","authenticated-orcid":false,"given":"Ken","family":"Raffenetti","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4422-2911","authenticated-orcid":false,"given":"Hui","family":"Zhou","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"given":"Pavan","family":"Balaji","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"Meta, Palo Alto, CA, USA"}]},{"given":"Min","family":"Si","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"Meta, Palo Alto, CA, USA"}]},{"given":"Abdelhalim","family":"Amer","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"Cerebras Systems, Sunnyvale, CA, USA"}]},{"given":"Shintaro","family":"Iwasaki","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"Meta, Palo Alto, CA, USA"}]},{"given":"Sangmin","family":"Seo","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"Klaytn Foundation, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7165-7591","authenticated-orcid":false,"given":"Giuseppe","family":"Congiu","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"NVIDIA, Santa Clara, CA, USA"}]},{"given":"Robert","family":"Latham","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]},{"given":"Lena","family":"Oden","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"FernUniversit\u00e4t in Hagen, Hagen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9740-4746","authenticated-orcid":false,"given":"Thomas","family":"Gillis","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"},{"name":"NVIDIA, Santa Clara, CA, USA"}]},{"given":"Rohit","family":"Zambre","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, CA, USA"},{"name":"University of California, Irvine, CA, USA"}]},{"given":"Kaiming","family":"Ouyang","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, CA, USA"},{"name":"University of California, Riverside, CA, USA"}]},{"given":"Charles","family":"Archer","sequence":"additional","affiliation":[{"name":"Cornelis Networks, Chesterbrook, PA, USA"},{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Wesley","family":"Bland","sequence":"additional","affiliation":[{"name":"Meta, Palo Alto, CA, USA"},{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Jithin","family":"Jose","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"},{"name":"Microsoft, Redmond, WA, USA"}]},{"given":"Sayantan","family":"Sur","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, CA, USA"},{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Hajime","family":"Fujita","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"},{"name":"Fastly, San Francisco, CA, USA"}]},{"given":"Dmitry","family":"Durnov","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Michael","family":"Chuvelev","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Gengbin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Alex","family":"Brooks","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Sagar","family":"Thapaliya","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Taru","family":"Doodi","sequence":"additional","affiliation":[{"name":"NVIDIA, Santa Clara, CA, USA"},{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Maria","family":"Garazan","sequence":"additional","affiliation":[{"name":"Intel Corporation, Santa Clara, CA, USA"}]},{"given":"Steve","family":"Oyanagi","sequence":"additional","affiliation":[{"name":"Hewlett Packard Enterprise, Palo Alto, CA, USA"}]},{"given":"Marc","family":"Snir","sequence":"additional","affiliation":[{"name":"University of Illinois Urbana-Champaign, Urbana, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5532-3048","authenticated-orcid":false,"given":"Rajeev","family":"Thakur","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory, Lemont, IL, USA"}]}],"member":"179","published-online":{"date-parts":[[2025,1,9]]},"reference":[{"unstructured":"ANL (2017) Center for exascale simulation for advanced reactors. https:\/\/github.com\/ANL-CESAR.","key":"bibr1-10943420241311608"},{"doi-asserted-by":"publisher","key":"bibr2-10943420241311608","DOI":"10.1007\/978-3-642-15646-5_4"},{"doi-asserted-by":"publisher","key":"bibr3-10943420241311608","DOI":"10.1109\/SC.2012.71"},{"doi-asserted-by":"publisher","key":"bibr4-10943420241311608","DOI":"10.1002\/cpe.4851"},{"doi-asserted-by":"publisher","key":"bibr5-10943420241311608","DOI":"10.1177\/10943420241265936"},{"doi-asserted-by":"publisher","key":"bibr6-10943420241311608","DOI":"10.1109\/CCGrid.2016.111"},{"doi-asserted-by":"publisher","key":"bibr7-10943420241311608","DOI":"10.1177\/1094342014548772"},{"unstructured":"DOE (2017) CORAL benchmarks. https:\/\/asc.llnl.gov\/CORAL-benchmarks.","key":"bibr8-10943420241311608"},{"unstructured":"ExaCT (2017) Center for exascale simulation of combustion in turbulence. https:\/\/www.exactcodesign.org\/.","key":"bibr9-10943420241311608"},{"unstructured":"Fischer PF, Lottes JW, Kerkemeier SG, et al. (2008) Nek5000: open source spectral element CFD solver. https:\/\/nek5000.mcs.anl.gov.","key":"bibr10-10943420241311608"},{"doi-asserted-by":"publisher","key":"bibr11-10943420241311608","DOI":"10.1109\/IPDPS.2017.18"},{"doi-asserted-by":"publisher","key":"bibr12-10943420241311608","DOI":"10.1145\/2503210.2504566"},{"doi-asserted-by":"publisher","key":"bibr13-10943420241311608","DOI":"10.1016\/j.jpdc.2020.05.008"},{"unstructured":"HPE (2024) HPE Slingshot interconnect. https:\/\/www.hpe.com\/us\/en\/compute\/hpc\/slingshot-interconnect.html.","key":"bibr14-10943420241311608"},{"unstructured":"Intel (2020) Intel Omni-Path fabric host software \u2013 user guide. https:\/\/www.intel.com\/content\/dam\/support\/us\/en\/documents\/network-and-i-o\/fabric-products\/Intel_OP_Fabric_Host_Software_UG_H76470_v16_0.pdf.","key":"bibr15-10943420241311608"},{"unstructured":"Intel (2024) Intel\u00ae software development emulator (Intel\u00ae SDE). https:\/\/www.intel.com\/content\/www\/us\/en\/developer\/articles\/tool\/software-development-emulator.html.","key":"bibr16-10943420241311608"},{"doi-asserted-by":"publisher","key":"bibr34-10943420241311608","DOI":"10.2172\/1090032"},{"doi-asserted-by":"publisher","key":"bibr18-10943420241311608","DOI":"10.1088\/1361-648X\/aab9c3"},{"doi-asserted-by":"publisher","key":"bibr19-10943420241311608","DOI":"10.1002\/cpe.3587"},{"unstructured":"MCB (2013) Monte Carlo Benchmark (MCB). https:\/\/codesign.llnl.gov\/mcb.php.","key":"bibr20-10943420241311608"},{"unstructured":"NASA (2017) NAS parallel benchmarks. https:\/\/www.nas.nasa.gov\/software\/npb.html.","key":"bibr21-10943420241311608"},{"unstructured":"Nekbone (2017) Nekbone. https:\/\/github.com\/Nek5000\/Nekbone.","key":"bibr22-10943420241311608"},{"unstructured":"NVIDIA (2016) Mellanox adapters programmer\u2019s reference manual. https:\/\/network.nvidia.com\/files\/doc-2020\/ethernet-adapters-programming-manual.pdf.","key":"bibr23-10943420241311608"},{"unstructured":"OFI-Working-Group (2024) Libfabric programmer\u2019s manual. https:\/\/ofiwg.github.io\/libfabric\/.","key":"bibr24-10943420241311608"},{"doi-asserted-by":"publisher","key":"bibr25-10943420241311608","DOI":"10.1109\/ICPP.2013.17"},{"doi-asserted-by":"publisher","key":"bibr26-10943420241311608","DOI":"10.1145\/3126908.3126963"},{"doi-asserted-by":"publisher","key":"bibr27-10943420241311608","DOI":"10.1007\/978-3-540-39924-7_55"},{"doi-asserted-by":"publisher","key":"bibr28-10943420241311608","DOI":"10.1016\/S0167-8191(98)00080-5"},{"doi-asserted-by":"publisher","key":"bibr29-10943420241311608","DOI":"10.1109\/HOTI55740.2022.00018"},{"unstructured":"UCF-Consortium (2024) Unified communication X. https:\/\/openucx.org.","key":"bibr30-10943420241311608"},{"doi-asserted-by":"publisher","key":"bibr31-10943420241311608","DOI":"10.1016\/j.cpc.2010.04.018"},{"doi-asserted-by":"publisher","key":"bibr32-10943420241311608","DOI":"10.1145\/3392717.3392773"},{"doi-asserted-by":"publisher","key":"bibr33-10943420241311608","DOI":"10.1145\/3555819.3555820"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241311608","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420241311608","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241311608","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241311608","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T18:14:09Z","timestamp":1741112049000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420241311608"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,9]]},"references-count":33,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["10.1177\/10943420241311608"],"URL":"https:\/\/doi.org\/10.1177\/10943420241311608","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"type":"print","value":"1094-3420"},{"type":"electronic","value":"1741-2846"}],"subject":[],"published":{"date-parts":[[2025,1,9]]}}}