{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T02:52:41Z","timestamp":1783997561720,"version":"3.55.0"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2011,10,1]],"date-time":"2011-10-01T00:00:00Z","timestamp":1317427200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE-AC02-Q6CH11357"],"award-info":[{"award-number":["DE-AC02-Q6CH11357"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006132","name":"Office of Science","doi-asserted-by":"publisher","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}],"id":[{"id":"10.13039\/100006132","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006192","name":"Advanced Scientific Computing Research","doi-asserted-by":"publisher","award":["DE-AC02-06CH11357"],"award-info":[{"award-number":["DE-AC02-06CH11357"]}],"id":[{"id":"10.13039\/100006192","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2011,10]]},"abstract":"<jats:p>Computational science applications are driving a demand for increasingly powerful storage systems. While many techniques are available for capturing the I\/O behavior of individual application trial runs and specific components of the storage system, continuous characterization of a production system remains a daunting challenge for systems with hundreds of thousands of compute cores and multiple petabytes of storage. As a result, these storage systems are often designed without a clear understanding of the diverse computational science workloads they will support.<\/jats:p>\n          <jats:p>In this study, we outline a methodology for scalable, continuous, systemwide I\/O characterization that combines storage device instrumentation, static file system analysis, and a new mechanism for capturing detailed application-level behavior. This methodology allows us to identify both system-wide trends and application-specific I\/O strategies. We demonstrate the effectiveness of our methodology by performing a multilevel, two-month study of Intrepid, a 557-teraflop IBM Blue Gene\/P system. During that time, we captured application-level I\/O characterizations from 6,481 unique jobs spanning 38 science and engineering projects. We used the results of our study to tune example applications, highlight trends that impact the design of future storage systems, and identify opportunities for improvement in I\/O characterization methodology.<\/jats:p>","DOI":"10.1145\/2027066.2027068","type":"journal-article","created":{"date-parts":[[2011,10,25]],"date-time":"2011-10-25T12:23:05Z","timestamp":1319545385000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":134,"title":["Understanding and Improving Computational Science Storage Access through Continuous Characterization"],"prefix":"10.1145","volume":"7","author":[{"given":"Philip","family":"Carns","sequence":"first","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Harms","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Allcock","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charles","family":"Bacon","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Lang","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Latham","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Ross","sequence":"additional","affiliation":[{"name":"Argonne National Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,10]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1453175.1453184"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1525908.1525919"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Workshop on Interfaces and Architectures for Scientific Data Storage.","author":"Carns P.","unstructured":"Carns , P. , Latham , R. , Ross , R. , Iskra , K. , Lang , S. , and Riley , K . 2009. 24\/7 characterization of petascale I\/O workloads . In Proceedings of the Workshop on Interfaces and Architectures for Scientific Data Storage. Carns, P., Latham, R., Ross, R., Iskra, K., Lang, S., and Riley, K. 2009. 24\/7 characterization of petascale I\/O workloads. In Proceedings of the Workshop on Interfaces and Architectures for Scientific Data Storage."},{"key":"e_1_2_1_4_1","unstructured":"Darshan. 2010. Darshan. http:\/\/www.mcs.anl.gov\/research\/projects\/darshan\/. Darshan . 2010. Darshan. http:\/\/www.mcs.anl.gov\/research\/projects\/darshan\/."},{"key":"e_1_2_1_5_1","volume-title":"Parallel Data Lab","author":"Dayal S.","unstructured":"Dayal , S. 2008. Characterizing HEC storage systems at rest. Tech. rep. CMU-PDL-08-109 , Parallel Data Lab , Carnegie Mellon University . Dayal, S. 2008. Characterizing HEC storage systems at rest. Tech. rep. CMU-PDL-08-109, Parallel Data Lab, Carnegie Mellon University."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the Computer Measurement Group (CMG) Conference. 1263--1269","author":"Ganger G. R.","year":"1995","unstructured":"Ganger , G. R. 1995 . Generating representative synthetic workloads: An unsolved problem . In Proceedings of the Computer Measurement Group (CMG) Conference. 1263--1269 . Ganger, G. R. 1995. Generating representative synthetic workloads: An unsolved problem. In Proceedings of the Computer Measurement Group (CMG) Conference. 1263--1269."},{"key":"e_1_2_1_7_1","unstructured":"Godard S. 2010. SYSSTATutilities homepage. http:\/\/pagesperso-orange.fr\/sebastien.godard\/. Godard S. 2010. SYSSTATutilities homepage. http:\/\/pagesperso-orange.fr\/sebastien.godard\/."},{"key":"e_1_2_1_8_1","unstructured":"INCITE. 2010. U.S. Department of Energy INCITE program. http:\/\/www.er.doe.gov\/ascr\/incite\/. INCITE . 2010. U.S. Department of Energy INCITE program. http:\/\/www.er.doe.gov\/ascr\/incite\/."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 5th Petascale Data Storage Workshop (PDSW). 1--5.","author":"Kim Y.","unstructured":"Kim , Y. , Gunasekaran , R. , Shipman , G. , Dillow , D. , Zhang , Z. , and Settlemyer , B . 2010. Workload characterization of a leadership class storage cluster . In Proceedings of the 5th Petascale Data Storage Workshop (PDSW). 1--5. Kim, Y., Gunasekaran, R., Shipman, G., Dillow, D., Zhang, Z., and Settlemyer, B. 2010. Workload characterization of a leadership class storage cluster. In Proceedings of the 5th Petascale Data Storage Workshop (PDSW). 1--5."},{"key":"e_1_2_1_10_1","volume-title":"SAND2008-3684","author":"Klundt R.","unstructured":"Klundt , R. , Weston , M. , and Ward , L . 2008. I\/O tracing on Catamount. Tech. rep . SAND2008-3684 , Sandia National Laboratory. Klundt, R., Weston, M., and Ward, L. 2008. I\/O tracing on Catamount. Tech. rep. SAND2008-3684, Sandia National Laboratory."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1374596.1374610"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1654059.1654100"},{"key":"e_1_2_1_13_1","unstructured":"LANL-Trace. 2010. HPC-5 open source software projects: LANL-Trace. http:\/\/institute.lanl.gov\/data\/software\/#lanl-trace. LANL-Trace . 2010. HPC-5 open source software projects: LANL-Trace. http:\/\/institute.lanl.gov\/data\/software\/#lanl-trace."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the USENIX Technical Conference. USENIX Association","author":"Leung A. W.","unstructured":"Leung , A. W. , Pasupathy , S. , Goodson , G. , and Miller , E. L . 2008. Measurement and analysis of large-scale network file system workloads . In Proceedings of the USENIX Technical Conference. USENIX Association , Berkeley, CA, 213--226. Leung, A. W., Pasupathy, S., Goodson, G., and Miller, E. L. 2008. Measurement and analysis of large-scale network file system workloads. In Proceedings of the USENIX Technical Conference. USENIX Association, Berkeley, CA, 213--226."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the ACM\/IEEE Conference on Supercomputing. IEEE Press","author":"Liao W.","unstructured":"Liao , W. and Choudhary , A . 2008. Dynamically adapting file domain partitioning methods for collective I\/O based on underlying parallel file system locking protocols . In Proceedings of the ACM\/IEEE Conference on Supercomputing. IEEE Press , Los Alamitos, CA. Liao, W. and Choudhary, A. 2008. Dynamically adapting file domain partitioning methods for collective I\/O based on underlying parallel file system locking protocols. In Proceedings of the ACM\/IEEE Conference on Supercomputing. IEEE Press, Los Alamitos, CA."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/71.539739"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2008.09.001"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the Scalable Parallel Libraries Conference. IEEE Computer Society, 104--113","author":"Reed D. A.","unstructured":"Reed , D. A. , Aydt , R. A. , Noe , R. J. , Roth , P. C. , Shields , K. A. , Schwartz , B. W. , and Tavera , L. F . 1993. Scalable performance analysis: The Pablo performance analysis environment . In Proceedings of the Scalable Parallel Libraries Conference. IEEE Computer Society, 104--113 . Reed, D. A., Aydt, R. A., Noe, R. J., Roth, P. C., Shields, K. A., Schwartz, B. W., and Tavera, L. F. 1993. Scalable performance analysis: The Pablo performance analysis environment. In Proceedings of the Scalable Parallel Libraries Conference. IEEE Computer Society, 104--113."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1374596.1374609"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the FAST Conference on File and Storage Technologies.","author":"Schmuck F.","unstructured":"Schmuck , F. and Haskin , R . 2002. GPFS: A shared-disk file system for large computing clusters . In Proceedings of the FAST Conference on File and Storage Technologies. Schmuck, F. and Haskin, R. 2002. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the FAST Conference on File and Storage Technologies."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the IEEE International Parallel and Distributed Processing Symposium.","author":"Seelam S.","unstructured":"Seelam , S. , Chung , I.-H. , Hong , D.-Y. , Wen , H.-F. , and Yu , H . 2008. Early experiences in application level I\/O tracing on Blue Gene systems . In Proceedings of the IEEE International Parallel and Distributed Processing Symposium. Seelam, S., Chung, I.-H., Hong, D.-Y., Wen, H.-F., and Yu, H. 2008. Early experiences in application level I\/O tracing on Blue Gene systems. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the Conference on Modelling Techniques and Tools for Computer Performance Evaluation. Lecture Notes in Computer Science","volume":"1245","author":"Smirni E.","unstructured":"Smirni , E. and Reed , D . 1997. Workload characterization of input\/output intensive parallel applications . In Proceedings of the Conference on Modelling Techniques and Tools for Computer Performance Evaluation. Lecture Notes in Computer Science , vol. 1245 . Springer-Verlag, 169--180. Smirni, E. and Reed, D. 1997. Workload characterization of input\/output intensive parallel applications. In Proceedings of the Conference on Modelling Techniques and Tools for Computer Performance Evaluation. Lecture Notes in Computer Science, vol. 1245. Springer-Verlag, 169--180."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367829.1367831"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium.","author":"Uselton A.","unstructured":"Uselton , A. , Hawison , M. , Wright , N. , Skinner , D. , Shalf , J. , Oliker , L. , Keen , N. , and Karavanic , K . 2010. Parallel I\/O performance: From events to ensembles . In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium. Uselton, A., Hawison, M., Wright, N., Skinner, D., Shalf, J., Oliker, L., Keen, N., and Karavanic, K. 2010. Parallel I\/O performance: From events to ensembles. In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/568014.379590"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1713072.1713080"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 21st IEEE\/12th NASA Goddard Conference on Mass Storage Systems and Technologies. 139--152","author":"Wang F.","unstructured":"Wang , F. , Xin , Q. , Hong , B. , Brandt , S. A. , Miller , E. L. , Long , D. D. E. , and Mclarty , T. T . 2004. File system workload analysis for large scale scientific computing applications . In Proceedings of the 21st IEEE\/12th NASA Goddard Conference on Mass Storage Systems and Technologies. 139--152 . Wang, F., Xin, Q., Hong, B., Brandt, S. A., Miller, E. L., Long, D. D. E., and Mclarty, T. T. 2004. File system workload analysis for large scale scientific computing applications. In Proceedings of the 21st IEEE\/12th NASA Goddard Conference on Mass Storage Systems and Technologies. 139--152."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 10th LCI International Conference on High-Performance Clustered Computing.","author":"Wright N. J.","unstructured":"Wright , N. J. , Pfeiffer , W. , and Snavely , A . 2009. Characterizing parallel scaling of scientific applications using IPM . In Proceedings of the 10th LCI International Conference on High-Performance Clustered Computing. Wright, N. J., Pfeiffer, W., and Snavely, A. 2009. Characterizing parallel scaling of scientific applications using IPM. In Proceedings of the 10th LCI International Conference on High-Performance Clustered Computing."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2011.50"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 12th International Symposium on High-Performance Computer Architecture.","author":"Yu H.","unstructured":"Yu , H. , Sahoo , R. K. , Howson , C. , Almasi , G. , Castanos , J. G. , Gupta , M. , Moreira , J. E. , Parker , J. J. , Engelsiepen , T. E. , Ross , R. , Thakur , R. , Latham , R. , and Gropp , W. D . 2006. High performance file I\/O for the BlueGene\/L supercomputer . In Proceedings of the 12th International Symposium on High-Performance Computer Architecture. Yu, H., Sahoo, R. K., Howson, C., Almasi, G., Castanos, J. G., Gupta, M., Moreira, J. E., Parker, J. J., Engelsiepen, T. E., Ross, R., Thakur, R., Latham, R., and Gropp, W. D. 2006. High performance file I\/O for the BlueGene\/L supercomputer. In Proceedings of the 12th International Symposium on High-Performance Computer Architecture."}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2027066.2027068","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2027066.2027068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:26:15Z","timestamp":1750278375000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2027066.2027068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,10]]},"references-count":30,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2011,10]]}},"alternative-id":["10.1145\/2027066.2027068"],"URL":"https:\/\/doi.org\/10.1145\/2027066.2027068","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"value":"1553-3077","type":"print"},{"value":"1553-3093","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,10]]},"assertion":[{"value":"2011-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}