{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T05:39:31Z","timestamp":1740980371744,"version":"3.38.0"},"reference-count":49,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2014,12,18]],"date-time":"2014-12-18T00:00:00Z","timestamp":1418860800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2015,2]]},"abstract":"<jats:p> Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input\/output (I\/O) performance. In this study, we introduce a memory-conscious collective I\/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I\/O accesses in intranode and internode layers, and determines I\/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I\/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I\/O can have a significant positive impact on scientific discovery productivity. <\/jats:p>","DOI":"10.1177\/1094342014561696","type":"journal-article","created":{"date-parts":[[2014,12,19]],"date-time":"2014-12-19T10:55:20Z","timestamp":1418986520000},"page":"21-36","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":2,"title":["Collective input\/output under memory constraints"],"prefix":"10.1177","volume":"29","author":[{"given":"Yin","family":"Lu","sequence":"first","affiliation":[{"name":"Computer Science Department, Texas Tech University, USA"}]},{"given":"Yong","family":"Chen","sequence":"additional","affiliation":[{"name":"Computer Science Department, Texas Tech University, USA"}]},{"given":"Yu","family":"Zhuang","sequence":"additional","affiliation":[{"name":"Computer Science Department, Texas Tech University, USA"}]},{"given":"Jialin","family":"Liu","sequence":"additional","affiliation":[{"name":"Computer Science Department, Texas Tech University, USA"}]},{"given":"Rajeev","family":"Thakur","sequence":"additional","affiliation":[{"name":"Mathematics and Computer Science Division, Argonne National Laboratory, USA"}]}],"member":"179","published-online":{"date-parts":[[2014,12,18]]},"reference":[{"key":"bibr1-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/1996130.1996137"},{"key":"bibr2-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTR.2009.5289188"},{"journal-title":"proceedings of PVM\/MPI","year":"2009","author":"Blas JG","key":"bibr3-1094342014561696"},{"key":"bibr4-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/1278480.1278667"},{"key":"bibr5-1094342014561696","unstructured":"Bryant RE (2007) Data-Intensive supercomputing: The case for DISC. Tech Report CMU-CS-07-128, Carnegie Mellon University School of Computer Science."},{"volume-title":"proceedings of the 4th annual Linux showcase and conference","year":"2000","author":"Carns PH","key":"bibr6-1094342014561696"},{"key":"bibr7-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2011.79"},{"volume-title":"proceedings of the ACM\/IEEE supercomputing conference (SC\u201908)","year":"2008","author":"Chen Y","key":"bibr8-1094342014561696"},{"volume-title":"proceedings of the IEEE international parallel and distributed processing symposium (IPDPS\u201911)","year":"2011","author":"Chen Y","key":"bibr9-1094342014561696"},{"key":"bibr10-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/IPPS.1999.760432"},{"key":"bibr11-1094342014561696","unstructured":"DOE Innovative and Novel Computational Impact on Theory and Experiment program. Available at: http:\/\/hpc.science.doe.gov\/ (accessed 20 May 2013)."},{"key":"bibr12-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.353"},{"key":"bibr13-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1177\/1094342010391989"},{"key":"bibr14-1094342014561696","first-page":"131","author":"Fryxell B","year":"2000","journal-title":"Astrophysical Journal Supplement"},{"volume-title":"Using MPI-2","year":"1999","author":"Gropp WD","key":"bibr15-1094342014561696"},{"key":"bibr16-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2008.43"},{"key":"bibr17-1094342014561696","unstructured":"HDF Group. Hierarchical Data Format, Version 5. The National Center for Supercomputing Applications. Available at: http:\/\/www.hdfgroup.org\/HDF5\/ (accessed 16 April 2013)."},{"key":"bibr18-1094342014561696","unstructured":"Hrothgar. Available at: http:\/\/hygd.hpcc.ttu.edu\/ganglia\/ (accessed 26 February 2013)."},{"key":"bibr19-1094342014561696","unstructured":"IOR benchmark (n.d.) Available at: https:\/\/computing.llnl.gov\/ (accessed 17 January 2013)."},{"key":"bibr20-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2010.127"},{"key":"bibr21-1094342014561696","first-page":"153","author":"Iskra K","year":"2008","journal-title":"proceedings of the 13th ACM SIGPLAN symposium on principles and practice of parallel programming"},{"key":"bibr22-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/244764.244766"},{"key":"bibr23-1094342014561696","first-page":"1","author":"Lang S","year":"2009","journal-title":"CLUSTER"},{"key":"bibr24-1094342014561696","first-page":"275","volume-title":"proceedings of the 1997 USENIX annual technical conference","author":"Lei H","year":"1997"},{"key":"bibr25-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/1048935.1050189"},{"key":"bibr26-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2007.370239"},{"key":"bibr27-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/1383529.1383533"},{"key":"bibr28-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/2491661.2481430"},{"key":"bibr29-1094342014561696","unstructured":"Lustre 2.0 Operations Manual. Available at: http:\/\/wiki.lustre.org\/manual\/LustreManual20_HTML\/index.html, 2011."},{"volume-title":"Parallel I\/O for High Performance Computing","year":"2001","author":"May J","key":"bibr30-1094342014561696"},{"key":"bibr31-1094342014561696","unstructured":"Message Passing Interface Forum (1996) MPI-2: Extensions to the Message-Passing Interface. Available at: http:\/\/www.mpi-forum.org\/docs\/docs.html (accessed 12 December 2012)."},{"key":"bibr32-1094342014561696","unstructured":"MPI-IO Test (fs_test) benchmark (n.d.) Available at: http:\/\/institutes.lanl.gov\/data\/software\/#mpi-io. (accessed 8 January 2013)."},{"volume-title":"International Conference for High Performance Computing, Networking, Storage and Analysis","year":"2008","author":"Nisar A","key":"bibr33-1094342014561696"},{"key":"bibr34-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/1366230.1366236"},{"key":"bibr35-1094342014561696","unstructured":"ROMIO website (n.d.) Available at: http:\/\/press3.mcs.anl.gov\/romio\/ (accessed 15 October 2012)."},{"key":"bibr36-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1145\/224170.224371"},{"key":"bibr37-1094342014561696","first-page":"231","volume-title":"proceedings of the first USENIX conference on file and storage technologies, USENIX","author":"Schmuck F","year":"2002"},{"key":"bibr38-1094342014561696","first-page":"1","author":"Shalf J","year":"2010","journal-title":"VECPAR"},{"key":"bibr39-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/FMPC.1996.558080"},{"key":"bibr40-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/FMPC.1999.750599"},{"key":"bibr41-1094342014561696","first-page":"95","volume-title":"high performance computing workshop","author":"Vetter JS","year":"2008"},{"key":"bibr42-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1007\/s10586-006-4895-y"},{"key":"bibr43-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2008.166"},{"volume-title":"proceedings of the USENIX symposium on operating systems design and implementation","year":"2006","author":"Weil S","key":"bibr44-1094342014561696"},{"volume-title":"proceedings of the 6th USENIX conference on file and storage technologies","year":"2008","author":"Welch B","key":"bibr45-1094342014561696"},{"key":"bibr46-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1177\/1094342010369117"},{"key":"bibr47-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2013.114"},{"key":"bibr48-1094342014561696","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2009.5161070"},{"key":"bibr49-1094342014561696","unstructured":"Zingale M (2001) FLASH I\/O benchmark routine \u2013 Parallel HDF 5, March. Available at: http:\/\/www.ucolick.org\/\u223czingale\/flash_benchmark_io\/ (accessed 12 January 2013)."}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342014561696","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342014561696","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342014561696","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T12:38:00Z","timestamp":1740919080000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342014561696"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,12,18]]},"references-count":49,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,2]]}},"alternative-id":["10.1177\/1094342014561696"],"URL":"https:\/\/doi.org\/10.1177\/1094342014561696","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"type":"print","value":"1094-3420"},{"type":"electronic","value":"1741-2846"}],"subject":[],"published":{"date-parts":[[2014,12,18]]}}}