{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,2]],"date-time":"2025-12-02T03:30:51Z","timestamp":1764646251816,"version":"3.41.0"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,5,18]],"date-time":"2020-05-18T00:00:00Z","timestamp":1589760000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"the National Science Foundation","award":["CCF 1725649"],"award-info":[{"award-number":["CCF 1725649"]}]},{"DOI":"10.13039\/100006234","name":"Sandia National Laboratories","doi-asserted-by":"publisher","award":["DE-NA-0003525"],"award-info":[{"award-number":["DE-NA-0003525"]}],"id":[{"id":"10.13039\/100006234","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Parallel Comput."],"published-print":{"date-parts":[[2020,6,30]]},"abstract":"<jats:p>The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and\/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. The experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.<\/jats:p>","DOI":"10.1145\/3391448","type":"journal-article","created":{"date-parts":[[2020,5,25]],"date-time":"2020-05-25T18:00:14Z","timestamp":1590429614000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["CoREC"],"prefix":"10.1145","volume":"7","author":[{"given":"Shaohua","family":"Duan","sequence":"first","affiliation":[{"name":"Rutgers Discovery Informatics Institute, Rutgers University, Piscataway, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pradeep","family":"Subedi","sequence":"additional","affiliation":[{"name":"Rutgers Discovery Informatics Institute, Rutgers University, Piscataway, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philip","family":"Davis","sequence":"additional","affiliation":[{"name":"Rutgers Discovery Informatics Institute, Rutgers University, Piscataway, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keita","family":"Teranishi","sequence":"additional","affiliation":[{"name":"Sandia National Laboratory, Livermore, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hemanth","family":"Kolla","sequence":"additional","affiliation":[{"name":"Sandia National Laboratory, Livermore, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marc","family":"Gamell","sequence":"additional","affiliation":[{"name":"Intel, Austin, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Manish","family":"Parashar","sequence":"additional","affiliation":[{"name":"Rutgers Discovery Informatics Institute, Rutgers University, Piscataway, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,5,18]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 10th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201910)","author":"Arturo Leonardo","year":"2010","unstructured":"Leonardo Arturo , Bautista Gomez , Naoya Maruyama , and Franck Cappello . 2010 . Distributed diskless checkpoint for large scale systems . In Proceedings of the 10th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201910) . 263--272. Leonardo Arturo, Bautista Gomez, Naoya Maruyama, and Franck Cappello. 2010. Distributed diskless checkpoint for large scale systems. In Proceedings of the 10th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201910). 263--272."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2019.00072"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2016.100"},{"volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201912)","author":"Bennett J. C.","key":"e_1_2_1_4_1","unstructured":"J. C. Bennett , H. Abbasi , P.-T. Bremer , R. Grout , A. Gyulassy , Tong Jin , S. Klasky , H. Kolla , M. Parashar , V. Pascucci , P. Pebay , D. Thompson , Hongfeng Yu , Fan Zhang , and J. Chen . 2012. Combining in-situ and in-transit processing to enable extreme-scale scientific analysis . In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201912) . 1--9. J. C. Bennett, H. Abbasi, P.-T. Bremer, R. Grout, A. Gyulassy, Tong Jin, S. Klasky, H. Kolla, M. Parashar, V. Pascucci, P. Pebay, D. Thompson, Hongfeng Yu, Fan Zhang, and J. Chen. 2012. Combining in-situ and in-transit processing to enable extreme-scale scientific analysis. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201912). 1--9."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.199"},{"key":"e_1_2_1_6_1","volume-title":"Manish Parashar, Stefan M. Wild, and H","author":"Bethel E. Wes","year":"2016","unstructured":"E. Wes Bethel , Martin Greenwald , Kersten Kleese van Dam , Manish Parashar, Stefan M. Wild, and H . Steven Wiley . 2016 . Report of the DOE workshop on management, analysis, and visualization of experimental and observational data\u2014The convergence of data and computing. In 2016 IEEE 12th International Conference on e-Science (e-Science). 213--222. E. Wes Bethel, Martin Greenwald, Kersten Kleese van Dam, Manish Parashar, Stefan M. Wild, and H. Steven Wiley. 2016. Report of the DOE workshop on management, analysis, and visualization of experimental and observational data\u2014The convergence of data and computing. In 2016 IEEE 12th International Conference on e-Science (e-Science). 213--222."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342013488238"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33518-1_24"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14529\/jsfi140101"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2017.149"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"J. H. Chen A. Choudhary B. de Supinski M. DeVries E. R. Hawkes S. Klasky W. K. Liao K. L. Ma J. Mellor-Crummey N. Podhorszki R. Sankaran S. Shende and C. S. Yoo. 2009. Terascale direct numerical simulations of turbulent combustion using S3D. Comput. Sci. Discov. 2 1 (2009).  J. H. Chen A. Choudhary B. de Supinski M. DeVries E. R. Hawkes S. Klasky W. K. Liao K. L. Ma J. Mellor-Crummey N. Podhorszki R. Sankaran S. Shende and C. S. Yoo. 2009. Terascale direct numerical simulations of turbulent combustion using S3D. Comput. Sci. Discov. 2 1 (2009).","DOI":"10.1088\/1749-4699\/2\/1\/015001"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201913)","author":"Cidon Asaf","year":"2013","unstructured":"Asaf Cidon , Ryan Stutsman , Stephen Rumble , Sachin Katti , John Ousterhout , and Mendel Rosenblum . 2013 . MinCopysets: Derandomizing replication in cloud storage . In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201913) . Asaf Cidon, Ryan Stutsman, Stephen Rumble, Sachin Katti, John Ousterhout, and Mendel Rosenblum. 2013. MinCopysets: Derandomizing replication in cloud storage. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201913)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1851476.1851481"},{"key":"e_1_2_1_14_1","volume-title":"DataSpaces: An interaction and coordination framework for coupled simulation workflows. Cluster Comput. 15, 2 (01","author":"Docan Ciprian","year":"2012","unstructured":"Ciprian Docan , Manish Parashar , and Scott Klasky . 2012. DataSpaces: An interaction and coordination framework for coupled simulation workflows. Cluster Comput. 15, 2 (01 June 2012 ), 163--181. Ciprian Docan, Manish Parashar, and Scott Klasky. 2012. DataSpaces: An interaction and coordination framework for coupled simulation workflows. Cluster Comput. 15, 2 (01 June 2012), 163--181."},{"key":"e_1_2_1_15_1","volume-title":"ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing. Concurr. Comput. Pract. Exper. 27, 14","author":"Docan Ciprian","year":"2014","unstructured":"Ciprian Docan , Fan Zhang , Tong Jin , Hoang Bui , Qian Sun , Julian Cummings , Norbert Podhorszki , Scott Klasky , and Manish Parashar . 2014. ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing. Concurr. Comput. Pract. Exper. 27, 14 ( 2014 ). Ciprian Docan, Fan Zhang, Tong Jin, Hoang Bui, Qian Sun, Julian Cummings, Norbert Podhorszki, Scott Klasky, and Manish Parashar. 2014. ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing. Concurr. Comput. Pract. Exper. 27, 14 (2014)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2018.00021"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-013-0884-0"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2012.56"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807672"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2751205.2751212"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 728--740","author":"Gomez Leonardo Bautista","year":"2012","unstructured":"Leonardo Bautista Gomez , Dimitri Komatitsch , and Naoya Maruyama . 2012 . FTI: High performance fault tolerance interface for hybrid systems . In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 728--740 . Leonardo Bautista Gomez, Dimitri Komatitsch, and Naoya Maruyama. 2012. FTI: High performance fault tolerance interface for hybrid systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 728--740."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126908.3126937"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2015.52"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2018.00046"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2016.7870922"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 17th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201917)","author":"Liu Jiaqi","year":"2017","unstructured":"Jiaqi Liu and Gagan Agrawal . 2017 . Supporting fault-tolerance in presence of in-situ analytics . In Proceedings of the 17th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201917) . 304--313. Jiaqi Liu and Gagan Agrawal. 2017. Supporting fault-tolerance in presence of in-situ analytics. In Proceedings of the 17th IEEE\/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid\u201917). 304--313."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/DISCS.2014.13"},{"volume-title":"Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA\u201916)","author":"Nie Bin","key":"e_1_2_1_28_1","unstructured":"Bin Nie , Devesh Tiwari , Saurabh Gupta , Evgenia Smirni , and James H. Rogers . 2016. A large-scale study of soft-errors on GPUs in the field . In Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA\u201916) . 519--530. Bin Nie, Devesh Tiwari, Saurabh Gupta, Evgenia Smirni, and James H. Rogers. 2016. A large-scale study of soft-errors on GPUs in the field. In Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA\u201916). 519--530."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2018.00022"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1137\/0108018"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807622"},{"volume-title":"Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST\u201909)","author":"Plank James S.","key":"e_1_2_1_32_1","unstructured":"James S. Plank , J. Luo , Catherine D. Schuman , L. Xu , and Z. Wilcox-O\u2019Hearn . 2009. A performance evaluation and examination of open-source erasure coding libraries for storage . In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST\u201909) . 263--272. James S. Plank, J. Luo, Catherine D. Schuman, L. Xu, and Z. Wilcox-O\u2019Hearn. 2009. A performance evaluation and examination of open-source erasure coding libraries for storage. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST\u201909). 263--272."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2018.00076"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2013.155"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2017.35"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST\u201913)","author":"Tiwari Devesh","year":"2013","unstructured":"Devesh Tiwari , Simona Boboila , Sudharshan S. Vazhkudai , Youngjae Kim , Xiaosong Ma , Peter J. Desnoyers , and Yan Solihin . 2013 . Active flash: Towards energy-efficient, in-situ data analytics on extreme-scale machines . In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST\u201913) . 119--132. Devesh Tiwari, Simona Boboila, Sudharshan S. Vazhkudai, Youngjae Kim, Xiaosong Ma, Peter J. Desnoyers, and Yan Solihin. 2013. Active flash: Towards energy-efficient, in-situ data analytics on extreme-scale machines. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST\u201913). 119--132."},{"key":"e_1_2_1_37_1","volume-title":"Office of Science","author":"U.S. Department of Energy","year":"2018","unstructured":"U.S. Department of Energy , Office of Science . 2018 . Exascale Computing Project. Retrieved from https:\/\/www.exascaleproject.org\/exascale-computing-project\/. U.S. Department of Energy, Office of Science. 2018. Exascale Computing Project. Retrieved from https:\/\/www.exascaleproject.org\/exascale-computing-project\/."},{"volume-title":"Proceedings of the 9th Workshop on Hot Topics in Dependable Systems. 263--272","author":"Vogt Dirk","key":"e_1_2_1_38_1","unstructured":"Dirk Vogt , Cristiano Giuffrida , Herbert Bos , and Andrew S. Tanenbaum . 2013. Techniques for efficient in-memory checkpointing . In Proceedings of the 9th Workshop on Hot Topics in Dependable Systems. 263--272 . Dirk Vogt, Cristiano Giuffrida, Herbert Bos, and Andrew S. Tanenbaum. 2013. Techniques for efficient in-memory checkpointing. In Proceedings of the 9th Workshop on Hot Topics in Dependable Systems. 263--272."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906)","author":"Weil Sage A.","year":"2006","unstructured":"Sage A. Weil , Scott A. Brandt , Ethan L. Miller , Darrell D. E. Long , and Carlos Maltzahn . 2006 . Ceph: A scalable, high-performance distributed file system . In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906) . USENIX Association, 307--320. Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI\u201906). USENIX Association, 307--320."},{"volume-title":"Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR\u201917)","author":"Yiu Matt M. T.","key":"e_1_2_1_40_1","unstructured":"Matt M. T. Yiu , Helen H. W. Chan , and Patrick P. C. Lee . 2017. Erasure coding for small objects in in-memory KV storage . In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR\u201917) . Matt M. T. Yiu, Helen H. W. Chan, and Patrick P. C. Lee. 2017. Erasure coding for small objects in in-memory KV storage. In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR\u201917)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MCG.2010.55","article-title":"In situ visualization for large-scale combustion simulations","volume":"3","author":"Yu Hongfeng","year":"2010","unstructured":"Hongfeng Yu , Chaoli Wang , Ray W. Grout , Jacqueline H. Chen , and Kwan-Liu Ma . 2010 . In situ visualization for large-scale combustion simulations . IEEE Comput. Graph. Applic. 3 (2010), 45 -- 57 . Hongfeng Yu, Chaoli Wang, Ray W. Grout, Jacqueline H. Chen, and Kwan-Liu Ma. 2010. In situ visualization for large-scale combustion simulations. IEEE Comput. Graph. Applic. 3 (2010), 45--57.","journal-title":"IEEE Comput. Graph. Applic."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/2930583.2930596"},{"volume-title":"Proceedings of the IEEE International Symposium on Parallel Distributed Processing (IPDPS\u201910)","author":"Zheng Fang","key":"e_1_2_1_43_1","unstructured":"Fang Zheng , H. Abbasi , C. Docan , J. Lofstead , Qing Liu , S. Klasky , M. Parashar , N. Podhorszki , K. Schwan , and M. Wolf . 2010. PreDatAPreparatory data analytics on peta-scale machines . In Proceedings of the IEEE International Symposium on Parallel Distributed Processing (IPDPS\u201910) . 1--12. Fang Zheng, H. Abbasi, C. Docan, J. Lofstead, Qing Liu, S. Klasky, M. Parashar, N. Podhorszki, K. Schwan, and M. Wolf. 2010. PreDatAPreparatory data analytics on peta-scale machines. In Proceedings of the IEEE International Symposium on Parallel Distributed Processing (IPDPS\u201910). 1--12."}],"container-title":["ACM Transactions on Parallel Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3391448","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3391448","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3391448","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:41Z","timestamp":1750200101000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3391448"}},"subtitle":["Scalable and Resilient In-memory Data Staging for In-situ Workflows"],"short-title":[],"issued":{"date-parts":[[2020,5,18]]},"references-count":43,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,30]]}},"alternative-id":["10.1145\/3391448"],"URL":"https:\/\/doi.org\/10.1145\/3391448","relation":{},"ISSN":["2329-4949","2329-4957"],"issn-type":[{"type":"print","value":"2329-4949"},{"type":"electronic","value":"2329-4957"}],"subject":[],"published":{"date-parts":[[2020,5,18]]},"assertion":[{"value":"2019-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}