{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T15:02:18Z","timestamp":1779375738564,"version":"3.53.1"},"reference-count":38,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T00:00:00Z","timestamp":1729036800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:p>Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I\/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I\/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I\/O library. The team developed an Asynchronous I\/O Virtual Object Layer (VOL) connector that allowed overlapping I\/O with computation. They also created a Cache VOL to complement asynchronous I\/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I\/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I\/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I\/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I\/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).<\/jats:p>","DOI":"10.1177\/10943420241288244","type":"journal-article","created":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T21:13:22Z","timestamp":1729113202000},"page":"65-78","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":6,"title":["HDF5 in the exascale era: Delivering efficient and scalable parallel I\/O for exascale applications"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9858-681X","authenticated-orcid":false,"given":"M","family":"Scot Breitenfeld","sequence":"first","affiliation":[{"name":"Engineering, The HDF Group, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Houjun","family":"Tang","sequence":"additional","affiliation":[{"name":"Scientific Data Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huihuo","family":"Zheng","sequence":"additional","affiliation":[{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jordan","family":"Henderson","sequence":"additional","affiliation":[{"name":"Engineering, The HDF Group, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suren","family":"Byna","sequence":"additional","affiliation":[{"name":"Scientific Data Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"},{"name":"Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2024,10,16]]},"reference":[{"key":"bibr1-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1145\/1551609.1551618"},{"key":"bibr2-10943420241288244","volume-title":"Polaris Supercomputer","author":"ALCF","year":"2023"},{"key":"bibr3-10943420241288244","volume-title":"ALCF Storage and Networking","author":"ALCF","year":"2024"},{"key":"bibr4-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/P3HPC49587.2019.00012"},{"key":"bibr5-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2954403"},{"key":"bibr6-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/180\/1\/012055"},{"key":"bibr7-10943420241288244","doi-asserted-by":"publisher","DOI":"10.2172\/964079"},{"key":"bibr8-10943420241288244","volume-title":"Tuning HDF5 Subfiling Performance on Parallel File Systems","author":"Byna S","year":"2023"},{"key":"bibr9-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1038\/s41578-023-00540-6"},{"key":"bibr10-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"bibr11-10943420241288244","unstructured":"Folk M, Cheng A, Yates K (1999) HDF5: a file format and I\/O library for high performance computing applications. In: Proceedings of supercomputing, Portland, OR, 14\u201319 November 1999, pp. 5\u201333."},{"key":"bibr12-10943420241288244","unstructured":"Foucar J (2024) Energy exascale earth system model project. URL: https:\/\/github.com\/E3SM-Project\/E3SM."},{"key":"bibr13-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2009.68"},{"key":"bibr14-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1016\/j.softx.2020.100561"},{"key":"bibr15-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/PDSW56643.2022.00008"},{"key":"bibr16-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/BigData52589.2021.9671638"},{"key":"bibr17-10943420241288244","unstructured":"Lee G, Shin S, Song W, et al. (2019) Asynchronous I\/O stack: a low-latency kernel I\/O stack for ultra-low latency SSDs. In: USENIX annual technical conference, Renton, WA, 10\u201312 July 2019, pp. 603\u2013616."},{"key":"bibr18-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2003.10053"},{"key":"bibr19-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-48842-0_3"},{"key":"bibr20-10943420241288244","unstructured":"Liao WK, Hou K (2023) Performance benchmark of E3SM\u2019s I\/O kernel. URL: https:\/\/github.com\/Parallel-NetCDF\/E3SM-IO."},{"key":"bibr21-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3125"},{"key":"bibr22-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2018.00068"},{"key":"bibr23-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1177\/8755293020970982"},{"key":"bibr24-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1177\/8755293020970980"},{"key":"bibr25-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1177\/10943420211019118"},{"key":"bibr26-10943420241288244","unstructured":"McCoy R (2020) Scorpio \u2013 parallel i\/o library. URL: https:\/\/e3sm.org\/scorpio-parallel-io-library\/."},{"key":"bibr27-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1177\/10943420211022829"},{"key":"bibr28-10943420241288244","volume-title":"Frontier user guide, OLCF user documentation","author":"ORNL","year":"2024"},{"key":"bibr29-10943420241288244","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.8322590"},{"key":"bibr30-10943420241288244","doi-asserted-by":"publisher","DOI":"10.21105\/joss.04115"},{"key":"bibr31-10943420241288244","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7672327"},{"key":"bibr32-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2018.00026"},{"key":"bibr33-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/PDSW49588.2019.00006"},{"key":"bibr34-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3090322"},{"key":"bibr35-10943420241288244","unstructured":"The HDF Group (2024) Hierarchical data format, version 5. URL: https:\/\/github.com\/HDFGroup\/hdf5."},{"key":"bibr36-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1115\/1.4064534"},{"key":"bibr37-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3100784"},{"key":"bibr39-10943420241288244","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid54584.2022.00015"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241288244","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420241288244","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420241288244","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:17:40Z","timestamp":1777450660000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420241288244"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,16]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10.1177\/10943420241288244"],"URL":"https:\/\/doi.org\/10.1177\/10943420241288244","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,16]]}}}