{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,6]],"date-time":"2025-12-06T16:45:07Z","timestamp":1765039507807,"version":"3.41.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,11,30]],"date-time":"2019-11-30T00:00:00Z","timestamp":1575072000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100011030","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE-AC05-00OR22725"],"award-info":[{"award-number":["DE-AC05-00OR22725"]}],"id":[{"id":"10.13039\/100011030","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2019,11,30]]},"abstract":"<jats:p>This article studies the I\/O write behaviors of the Titan supercomputer and its Lustre parallel file stores under production load. The results can inform the design, deployment, and configuration of file systems along with the design of I\/O software in the application, operating system, and adaptive I\/O libraries.<\/jats:p>\n          <jats:p>We propose a statistical benchmarking methodology to measure write performance across I\/O configurations, hardware settings, and system conditions. Moreover, we introduce two relative measures to quantify the write-performance behaviors of hardware components under production load. In addition to designing experiments and benchmarking on Titan, we verify the experimental results on one real application and one real application I\/O kernel, XGC and HACC IO, respectively. These two are representative and widely used to address the typical I\/O behaviors of applications.<\/jats:p>\n          <jats:p>In summary, we find that Titan\u2019s I\/O system is variable across the machine at fine time scales. This variability has two major implications. First, stragglers lessen the benefit of coupled I\/O parallelism (striping). Peak median output bandwidths are obtained with parallel writes to many independent files, with no striping or write sharing of files across clients (compute nodes). I\/O parallelism is most effective when the application\u2014or its I\/O libraries\u2014distributes the I\/O load so that each target stores files for multiple clients and each client writes files on multiple targets in a balanced way with minimal contention. Second, our results suggest that the potential benefit of dynamic adaptation is limited. In particular, it is not fruitful to attempt to identify \u201cgood locations\u201d in the machine or in the file system: component performance is driven by transient load conditions and past performance is not a useful predictor of future performance. For example, we do not observe diurnal load patterns that are predictable.<\/jats:p>","DOI":"10.1145\/3335205","type":"journal-article","created":{"date-parts":[[2020,4,4]],"date-time":"2020-04-04T10:48:51Z","timestamp":1585997331000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Characterizing Output Bottlenecks of a Production Supercomputer"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5409-4378","authenticated-orcid":false,"given":"Bing","family":"Xie","sequence":"first","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"Sarp","family":"Oral","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"Christopher","family":"Zimmer","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"Jong Youl","family":"Choi","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"David","family":"Dillow","sequence":"additional","affiliation":[{"name":"dave@thedillows.org"}]},{"given":"Scott","family":"Klasky","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"Jay","family":"Lofstead","sequence":"additional","affiliation":[{"name":"Sandia National Laboratories, Eubank SE, Albuquerque, NM"}]},{"given":"Norbert","family":"Podhorszki","sequence":"additional","affiliation":[{"name":"Oak Ridge National Laboratory, Oak Ridge, TN"}]},{"given":"Jeffrey S.","family":"Chase","sequence":"additional","affiliation":[{"name":"Duke University, Durham, NC"}]}],"member":"320","published-online":{"date-parts":[[2020,1,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Argonne National Laboratory. 2018. Retrieved November 9 2019 from Darshan: HPC I\/O Characterization Tool. http:\/\/www.mcs.anl.gov\/research\/projects\/darshan.  Argonne National Laboratory. 2018. Retrieved November 9 2019 from Darshan: HPC I\/O Characterization Tool. http:\/\/www.mcs.anl.gov\/research\/projects\/darshan."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2027066.2027068"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTR.2009.5289150"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2004.08.005"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1063\/1.2937116"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"J. H. Chen A. Choudhary B. de Supinski M. DeVries E. R. Hawkes S. Klasky W. Liao K. Ma J. Mellor-Crummey N. Podhorszki R. Sankaran S. Shende and C. Yoo. 2009. Terascale direct numerical simulations of turbulent combustion using S3D. Computational Science 8 Discovery 2 1 015001.  J. H. Chen A. Choudhary B. de Supinski M. DeVries E. R. Hawkes S. Klasky W. Liao K. Ma J. Mellor-Crummey N. Podhorszki R. Sankaran S. Shende and C. Yoo. 2009. Terascale direct numerical simulations of turbulent combustion using S3D. Computational Science 8 Discovery 2 1 015001.","DOI":"10.1088\/1749-4699\/2\/1\/015001"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043556.2043562"},{"volume-title":"Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201910)","author":"Cui Y.","key":"e_1_2_1_8_1","unstructured":"Y. Cui , K. Olsen , T. Jordan , K. Lee , J. Zhou , P. Small , D. Roten , G. Ely , D. K. Panda , A. Chourasia , J. Levesque , S. Day , and P. Maechling . 2010. Scalable earthquake simulation on petascale supercomputers . In Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201910) . Washington, DC. 1--20. Y. Cui, K. Olsen, T. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D. K. Panda, A. Chourasia, J. Levesque, S. Day, and P. Maechling. 2010. Scalable earthquake simulation on petascale supercomputers. In Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201910). Washington, DC. 1--20."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCCC.2011.6108062"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2014.56"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the Cray User Group Conference (CUG\u201914)","author":"Ezell Matt","year":"2014","unstructured":"Matt Ezell , David Dillow , Sarp Oral , Feiyi Wang , Devesh Tiwari , Don Maxwell , Dustin Leverman , and Jason Hill . 2014 . I\/O router placement and fine-grained routing on Titan to support Spider II . In Proceedings of the Cray User Group Conference (CUG\u201914) . Lugano, Switzerland, 1--6. Matt Ezell, David Dillow, Sarp Oral, Feiyi Wang, Devesh Tiwari, Don Maxwell, Dustin Leverman, and Jason Hill. 2014. I\/O router placement and fine-grained routing on Titan to support Spider II. In Proceedings of the Cray User Group Conference (CUG\u201914). Lugano, Switzerland, 1--6."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-014-1321-8"},{"volume-title":"Proceedings of the 5th Petascale Data Storage Workshop (PDSW\u201910)","author":"Kim Youngjae","key":"e_1_2_1_13_1","unstructured":"Youngjae Kim , Raghul Gunasekaran , Galen M. Shipman , David A. Dillow , Zhe Zhang , and Bradley W. Settlemyer . 2010. Workload characterization of a leadership class storage cluster . In Proceedings of the 5th Petascale Data Storage Workshop (PDSW\u201910) . New Orleans, LA, 1--5. Youngjae Kim, Raghul Gunasekaran, Galen M. Shipman, David A. Dillow, Zhe Zhang, and Bradley W. Settlemyer. 2010. Workload characterization of a leadership class storage cluster. In Proceedings of the 5th Petascale Data Storage Workshop (PDSW\u201910). New Orleans, LA, 1--5."},{"volume-title":"Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201903)","author":"Klasky S.","key":"e_1_2_1_14_1","unstructured":"S. Klasky , S. Ethier , Z. Lin , K. Martins , D. McCune , and R. Samtaney . 2003. Grid-based parallel data streaming implemented for the Gyrokinetic Toroidal Code . In Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201903) . Phoenix, AZ, 24--36. S. Klasky, S. Ethier, Z. Lin, K. Martins, D. McCune, and R. Samtaney. 2003. Grid-based parallel data streaming implemented for the Gyrokinetic Toroidal Code. In Proceedings of the ACM\/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201903). Phoenix, AZ, 24--36."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/214419.214421"},{"key":"e_1_2_1_16_1","first-page":"87","article-title":"Gyrokinetic particle simulation of neoclassical transport in the pedestal\/scrape-off region of a tokamak plasma","volume":"46","author":"Ku S.","year":"2006","unstructured":"S. Ku , C. S. Chang , M. Adams , J. Cummings , F. Hinton , D. Keyes , S. Klasky , W. Lee , Z. Lin , S. Parker , and the CPES team. 2006 . Gyrokinetic particle simulation of neoclassical transport in the pedestal\/scrape-off region of a tokamak plasma . Journal of Physics 46 , 1, 87 -- 91 . S. Ku, C. S. Chang, M. Adams, J. Cummings, F. Hinton, D. Keyes, S. Klasky, W. Lee, Z. Lin, S. Parker, and the CPES team. 2006. Gyrokinetic particle simulation of neoclassical transport in the pedestal\/scrape-off region of a tokamak plasma. Journal of Physics 46, 1, 87--91.","journal-title":"Journal of Physics"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-20119-1_19"},{"volume-title":"Proceedings of the ACM\/IEEE International Conference for High Performance Computing Networking, Storage and Analysis (SC\u201909)","author":"Lang S.","key":"e_1_2_1_18_1","unstructured":"S. Lang , P. Carns , R. Latham , R. Ross , K. Harms , and W. Allcock . 2009. I\/O performance challenges at leadership scale . In Proceedings of the ACM\/IEEE International Conference for High Performance Computing Networking, Storage and Analysis (SC\u201909) . Portland, OR, 40--52. S. Lang, P. Carns, R. Latham, R. Ross, K. Harms, and W. Allcock. 2009. I\/O performance challenges at leadership scale. In Proceedings of the ACM\/IEEE International Conference for High Performance Computing Networking, Storage and Analysis (SC\u201909). Portland, OR, 40--52."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3125"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2009.5161052"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2010.32"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2749246.2749269"},{"volume-title":"Proceedings of the International Conference on High Performance Computing","author":"Madireddy Sandeep","key":"e_1_2_1_23_1","unstructured":"Sandeep Madireddy , Prasanna Balaprakash , Phil Carns , Robert Latham , Robert Ross , Shane Snyder , and Stefan M. Wild . 2018. Machine learning based parallel I\/O predictive modeling: A case study on Lustre file systems . In Proceedings of the International Conference on High Performance Computing . Hyderabad, India, 184--204. Sandeep Madireddy, Prasanna Balaprakash, Phil Carns, Robert Latham, Robert Ross, Shane Snyder, and Stefan M. Wild. 2018. Machine learning based parallel I\/O predictive modeling: A case study on Lustre file systems. In Proceedings of the International Conference on High Performance Computing. Hyderabad, India, 184--204."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2016.58"},{"volume-title":"Oak Ridge Interconnect Workshop (ASCI-00-003","author":"David","key":"e_1_2_1_25_1","unstructured":"David A. Nowark and Mark Seager. 1999. ASCI terascale simulation: Requirements and deployments . In Oak Ridge Interconnect Workshop (ASCI-00-003 .1). Oak Ridge, TN, 1--15. David A. Nowark and Mark Seager. 1999. ASCI terascale simulation: Requirements and deployments. In Oak Ridge Interconnect Workshop (ASCI-00-003.1). Oak Ridge, TN, 1--15."},{"key":"e_1_2_1_26_1","unstructured":"Oak Ridge National Laboratory. 2018. HACC. Retrieved November 9 2019 from https:\/\/www.olcf.ornl.gov\/caar\/hacc\/.  Oak Ridge National Laboratory. 2018. HACC. Retrieved November 9 2019 from https:\/\/www.olcf.ornl.gov\/caar\/hacc\/."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855511.1855522"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2008.5222721"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the Cray User Group Meeting (CUG\u201907)","author":"Shan Hongzhang","year":"2007","unstructured":"Hongzhang Shan and John Shalf . 2007 . Using IOR to analyze the I\/O performance for HPC platforms . In Proceedings of the Cray User Group Meeting (CUG\u201907) . Washington, DC, 1--15. Hongzhang Shan and John Shalf. 2007. Using IOR to analyze the I\/O performance for HPC platforms. In Proceedings of the Cray User Group Meeting (CUG\u201907). Washington, DC, 1--15."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Cray User Group Conference (CUG\u201912)","author":"Shipman Galen","year":"2012","unstructured":"Galen Shipman , David Dillow , Douglas Fuller , Raghul Gunasekaran , Jason Hill , Youngjae Kim , Sarp Oral , Doug Reitz , James Simmons , and Feiyi Wang . 2012 . A next-generation parallel file system environment for the OLCF . In Proceedings of the Cray User Group Conference (CUG\u201912) . Stuttgart, Germany, 1--12. Galen Shipman, David Dillow, Douglas Fuller, Raghul Gunasekaran, Jason Hill, Youngjae Kim, Sarp Oral, Doug Reitz, James Simmons, and Feiyi Wang. 2012. A next-generation parallel file system environment for the OLCF. In Proceedings of the Cray User Group Conference (CUG\u201912). Stuttgart, Germany, 1--12."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the Cray User Group Meeting (CUG\u201909)","author":"Shipman Galen","year":"2009","unstructured":"Galen Shipman , David Dillow , Sarp Oral , and Feiyi Wang . 2009 . The Spider center wide file system: from concept to reality . In Proceedings of the Cray User Group Meeting (CUG\u201909) . Atlanta GA, 1--10. Galen Shipman, David Dillow, Sarp Oral, and Feiyi Wang. 2009. The Spider center wide file system: from concept to reality. In Proceedings of the Cray User Group Meeting (CUG\u201909). Atlanta GA, 1--10."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/795668.796733"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2011.18"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2010.5470424"},{"key":"e_1_2_1_35_1","volume-title":"George Ostrouchov, and Scott Klasky.","author":"Wan Lipeng","year":"2017","unstructured":"Lipeng Wan , Matthew Wolf , Feiyi Wang , Jong Youl Choi , George Ostrouchov, and Scott Klasky. 2017 . Analysis and modeling of the end-to-end I\/O performance on OLCF\u2019s titan supercomputer. In Proceedings of the 19th IEEE International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS\u201917). Salt Lake City, Utah , 1--9. Lipeng Wan, Matthew Wolf, Feiyi Wang, Jong Youl Choi, George Ostrouchov, and Scott Klasky. 2017. Analysis and modeling of the end-to-end I\/O performance on OLCF\u2019s titan supercomputer. In Proceedings of the 19th IEEE International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS\u201917). Salt Lake City, Utah, 1--9."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906)","author":"Weil Sage A.","year":"2006","unstructured":"Sage A. Weil , Scott A. Brandt , Ethan L. Miller , Darrell Long , and Carlos Maltzahn . 2006 . Ceph: A scalable, high-performance distributed file system . In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906) . Seattle, WA, 307--320. Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906). Seattle, WA, 307--320."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/2388996.2389007"},{"volume-title":"HPC I\/O in the Data Center Workshop (HPC-IODC\u201917).","author":"Xie Bing","key":"e_1_2_1_40_1","unstructured":"Bing Xie , Jeffrey S. Chase , David Dillow , Scott Klasky , Jay Lofstead , Sarp Oral , and Norbert Podhorszki . 2017. Output performance study on a production petascale filesystem . In HPC I\/O in the Data Center Workshop (HPC-IODC\u201917). Frankfurt, Germany , 1--14. Bing Xie, Jeffrey S. Chase, David Dillow, Scott Klasky, Jay Lofstead, Sarp Oral, and Norbert Podhorszki. 2017. Output performance study on a production petascale filesystem. In HPC I\/O in the Data Center Workshop (HPC-IODC\u201917). Frankfurt, Germany, 1--14."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078597.3078614"}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3335205","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3335205","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3335205","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:26:18Z","timestamp":1750206378000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3335205"}},"subtitle":["Analysis and Implications"],"short-title":[],"issued":{"date-parts":[[2019,11,30]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,11,30]]}},"alternative-id":["10.1145\/3335205"],"URL":"https:\/\/doi.org\/10.1145\/3335205","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"type":"print","value":"1553-3077"},{"type":"electronic","value":"1553-3093"}],"subject":[],"published":{"date-parts":[[2019,11,30]]},"assertion":[{"value":"2018-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}