{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:23:42Z","timestamp":1750220622916,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,1,15]],"date-time":"2020-01-15T00:00:00Z","timestamp":1579046400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,1,15]]},"DOI":"10.1145\/3368474.3368482","type":"proceedings-article","created":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T13:31:42Z","timestamp":1576589502000},"page":"150-160","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Exploiting Spark for HPC Simulation Data"],"prefix":"10.1145","author":[{"given":"Ming","family":"Jiang","sequence":"first","affiliation":[{"name":"Lawrence Livermore National, Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brian","family":"Gallagher","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National, Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Albert","family":"Chu","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National, Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ghaleb","family":"Abdulla","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National, Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timothy","family":"Bender","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National, Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,1,15]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2010. The Opportunities and Challenges of Exascale Computing ASCAC Subcommittee Report. http:\/\/science.energy.gov\/~\/media\/ascr\/ascac\/pdf\/reports\/Exascale_subcommittee_report.pdf.  2010. The Opportunities and Challenges of Exascale Computing ASCAC Subcommittee Report. http:\/\/science.energy.gov\/~\/media\/ascr\/ascac\/pdf\/reports\/Exascale_subcommittee_report.pdf."},{"key":"e_1_3_2_1_2_1","unstructured":"2012. Report out from the Exascale Research Planning Workshop Working Session on Data Management Visualization IO and Storage. http:\/\/exascaleresearch.labworks.org\/apr2012planningworkshop\/application\/layouts\/exascale-planning-workshop\/\/public\/docs\/PRES_WorkingSession-DataIO_120420.pdf.  2012. Report out from the Exascale Research Planning Workshop Working Session on Data Management Visualization IO and Storage. http:\/\/exascaleresearch.labworks.org\/apr2012planningworkshop\/application\/layouts\/exascale-planning-workshop\/\/public\/docs\/PRES_WorkingSession-DataIO_120420.pdf."},{"key":"e_1_3_2_1_3_1","unstructured":"2019. Hadoop. https:\/\/hadoop.apache.org\/  2019. Hadoop. https:\/\/hadoop.apache.org\/"},{"key":"e_1_3_2_1_4_1","unstructured":"2019. hanythingondemand. https:\/\/github.com\/hpcugent\/hanythingondemand  2019. hanythingondemand. https:\/\/github.com\/hpcugent\/hanythingondemand"},{"key":"e_1_3_2_1_5_1","unstructured":"2019. Magpie. https:\/\/github.com\/LLNL\/magpie  2019. Magpie. https:\/\/github.com\/LLNL\/magpie"},{"key":"e_1_3_2_1_6_1","unstructured":"2019. Sort Benchmark. http:\/\/sortbenchmark.org\/  2019. Sort Benchmark. http:\/\/sortbenchmark.org\/"},{"key":"e_1_3_2_1_7_1","unstructured":"2019. Spark. https:\/\/spark.apache.org\/  2019. Spark. https:\/\/spark.apache.org\/"},{"key":"e_1_3_2_1_8_1","unstructured":"2019. sparkhpc. https:\/\/github.com\/rokroskar\/sparkhpc  2019. sparkhpc. https:\/\/github.com\/rokroskar\/sparkhpc"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.36"},{"volume-title":"XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. Article 34","author":"Baer T.","key":"e_1_3_2_1_10_1","unstructured":"T. Baer , P. Peltz , J. Yin , and E. Begoli . 2015. Integrating Apache Spark into PBS-Based HPC Environments . In XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. Article 34 . T. Baer, P. Peltz, J. Yin, and E. Begoli. 2015. Integrating Apache Spark into PBS-Based HPC Environments. In XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure. Article 34."},{"key":"e_1_3_2_1_11_1","unstructured":"A. Cachuan. 2019. How does Facebook tune Apache Spark for Large-Scale Workloads? https:\/\/towardsdatascience.com\/how-does-facebook-tune-apache-spark-for-large-scale-workloads-3238ddda0830  A. Cachuan. 2019. How does Facebook tune Apache Spark for Large-Scale Workloads? https:\/\/towardsdatascience.com\/how-does-facebook-tune-apache-spark-for-large-scale-workloads-3238ddda0830"},{"volume-title":"Scaling Spark on HPC Systems. In ACM International Symposium on High-Performance Parallel and Distributed Computing. 97--110","author":"Chaimov N.","key":"e_1_3_2_1_12_1","unstructured":"N. Chaimov , A. Malony , S. Canon , C. Iancu , K. Ibrahim , and J. Srinivasan . 2016 . Scaling Spark on HPC Systems. In ACM International Symposium on High-Performance Parallel and Distributed Computing. 97--110 . N. Chaimov, A. Malony, S. Canon, C. Iancu, K. Ibrahim, and J. Srinivasan. 2016. Scaling Spark on HPC Systems. In ACM International Symposium on High-Performance Parallel and Distributed Computing. 97--110."},{"volume-title":"Towards Selecting Best Combination of SQL-on-Hadoop Systems and JVMs. In IEEE International Conference on Cloud Computing. 245--252","author":"Chiba T.","key":"e_1_3_2_1_13_1","unstructured":"T. Chiba , T. Yoshimura , M. Horie , and H. Horii . 2018 . Towards Selecting Best Combination of SQL-on-Hadoop Systems and JVMs. In IEEE International Conference on Cloud Computing. 245--252 . T. Chiba, T. Yoshimura, M. Horie, and H. Horii. 2018. Towards Selecting Best Combination of SQL-on-Hadoop Systems and JVMs. In IEEE International Conference on Cloud Computing. 245--252."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.02.026"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342019852127"},{"volume-title":"20 Years of Beowulf: Workshop to Honor of Thomas Sterling's 65th Birthday. 7--16","author":"Fox G.","key":"e_1_3_2_1_16_1","unstructured":"G. Fox , S. Jha , J. Qiu , and A. Luckow . 2014. Towards an Understanding of Facets and Exemplars of Big Data Applications . In 20 Years of Beowulf: Workshop to Honor of Thomas Sterling's 65th Birthday. 7--16 . G. Fox, S. Jha, J. Qiu, and A. Luckow. 2014. Towards an Understanding of Facets and Exemplars of Big Data Applications. In 20 Years of Beowulf: Workshop to Honor of Thomas Sterling's 65th Birthday. 7--16."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2016.05.297"},{"volume-title":"IEEE International Conference on Big Data. 2087--2096","author":"Harney J.","key":"e_1_3_2_1_18_1","unstructured":"J. Harney , S. Lim , S. Sukumar , D. Stansberry , and P. Xenopoulos . 2016. On-Demand Data Analytics in HPC Environments at Leadership Computing Facilities: Challenges and Experiences . In IEEE International Conference on Big Data. 2087--2096 . J. Harney, S. Lim, S. Sukumar, D. Stansberry, and P. Xenopoulos. 2016. On-Demand Data Analytics in HPC Environments at Leadership Computing Facilities: Challenges and Experiences. In IEEE International Conference on Big Data. 2087--2096."},{"volume-title":"Performance Characterization and Acceleration of In-Memory File Systems for Hadoop and Spark Applications on HPC Clusters. In IEEE International Conference on Big Data. 243--252","author":"Islam N.","key":"e_1_3_2_1_19_1","unstructured":"N. Islam , M. Wasi-ur- Rahman , X. Lu , D. Shankar , and D. Panda . 2015 . Performance Characterization and Acceleration of In-Memory File Systems for Hadoop and Spark Applications on HPC Clusters. In IEEE International Conference on Big Data. 243--252 . N. Islam, M. Wasi-ur-Rahman, X. Lu, D. Shankar, and D. Panda. 2015. Performance Characterization and Acceleration of In-Memory File Systems for Hadoop and Spark Applications on HPC Clusters. In IEEE International Conference on Big Data. 243--252."},{"volume-title":"IEEE International Conference on Machine Learning and Applications. 977--982","author":"Jiang M.","key":"e_1_3_2_1_20_1","unstructured":"M. Jiang , B. Gallagher , J. Kallman , and D. Laney . 2016. A Supervised Learning Framework for Arbitrary Lagrangian-Eulerian Simulations . In IEEE International Conference on Machine Learning and Applications. 977--982 . M. Jiang, B. Gallagher, J. Kallman, and D. Laney. 2016. A Supervised Learning Framework for Arbitrary Lagrangian-Eulerian Simulations. In IEEE International Conference on Machine Learning and Applications. 977--982."},{"key":"e_1_3_2_1_21_1","first-page":"168","article-title":"A Deep Learning Framework for Mesh Relaxation in Arbitrary Lagrangian-Eulerian Simulations","volume":"11139","author":"Jiang M.","year":"2019","unstructured":"M. Jiang , B. Gallagher , N. Mandell , A. Maguire , K. Henderson , and G. Weinert . 2019 . A Deep Learning Framework for Mesh Relaxation in Arbitrary Lagrangian-Eulerian Simulations . In SPIE Applications of Machine Learning , Vol. 11139. 168 -- 182 . M. Jiang, B. Gallagher, N. Mandell, A. Maguire, K. Henderson, and G. Weinert. 2019. A Deep Learning Framework for Mesh Relaxation in Arbitrary Lagrangian-Eulerian Simulations. In SPIE Applications of Machine Learning, Vol. 11139. 168--182.","journal-title":"SPIE Applications of Machine Learning"},{"key":"e_1_3_2_1_22_1","unstructured":"S. Krishnan M. Tatineni and C. Baru. 2011. myHadoop---Hadoop-on-Demand on Traditional HPC Resources. https:\/\/pdfs.semanticscholar.org\/1357\/8e78620db19f8185060f206a95b495435cc8.pdf  S. Krishnan M. Tatineni and C. Baru. 2011. myHadoop---Hadoop-on-Demand on Traditional HPC Resources. https:\/\/pdfs.semanticscholar.org\/1357\/8e78620db19f8185060f206a95b495435cc8.pdf"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2016.05.003"},{"volume-title":"IEEE Symposium on Large Data Analysis and Visualization. 55--63","author":"Ling J.","key":"e_1_3_2_1_24_1","unstructured":"J. Ling , W. Kegelmeyer , K. Aditya , H. Kolla , K. Reed , T. Shead , and W. Davis . 2017. Using Feature Importance Metrics to Detect Events of Interest in Scientific Computing Applications . In IEEE Symposium on Large Data Analysis and Visualization. 55--63 . J. Ling, W. Kegelmeyer, K. Aditya, H. Kolla, K. Reed, T. Shead, and W. Davis. 2017. Using Feature Importance Metrics to Detect Events of Interest in Scientific Computing Applications. In IEEE Symposium on Large Data Analysis and Visualization. 55--63."},{"key":"e_1_3_2_1_25_1","unstructured":"J. Liu E. Racah Q. Koziol and R. Canon. 2016. H5spark: Bridging the I\/O Gap Between Spark and Scientific Data Formats on HPC Systems. Cray User Group.  J. Liu E. Racah Q. Koziol and R. Canon. 2016. H5spark: Bridging the I\/O Gap Between Spark and Scientific Data Formats on HPC Systems. Cray User Group."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"N. Malitsky. 2016. Bringing the HPC Reconstruction Algorithms to Big Data Platforms. In Scientific Data Summit. 1--8.  N. Malitsky. 2016. Bringing the HPC Reconstruction Algorithms to Big Data Platforms. In Scientific Data Summit. 1--8.","DOI":"10.1109\/NYSDS.2016.7747818"},{"key":"e_1_3_2_1_27_1","unstructured":"S. Michael A. Thota and R. Henschel. 2014. HPCHadoop: A Framework to Run Hadoop on Cray X-Series Supercomputers. In Cray User Group.  S. Michael A. Thota and R. Henschel. 2014. HPCHadoop: A Framework to Run Hadoop on Cray X-Series Supercomputers. In Cray User Group."},{"volume-title":"Understanding the Influence of Configuration Settings: An Execution Model-Driven Framework for Apache Spark Platform. In IEEE International Conference on Cloud Computing. 802--807","author":"Nguyen N.","key":"e_1_3_2_1_28_1","unstructured":"N. Nguyen , M. Khan , Y. Albayram , and K. Wang . 2017 . Understanding the Influence of Configuration Settings: An Execution Model-Driven Framework for Apache Spark Platform. In IEEE International Conference on Cloud Computing. 802--807 . N. Nguyen, M. Khan, Y. Albayram, and K. Wang. 2017. Understanding the Influence of Configuration Settings: An Execution Model-Driven Framework for Apache Spark Platform. In IEEE International Conference on Cloud Computing. 802--807."},{"volume-title":"Towards Automatic Tuning of Apache Spark Configuration. In IEEE International Conference on Cloud Computing. 417--425","author":"Nguyen N.","key":"e_1_3_2_1_29_1","unstructured":"N. Nguyen , M. Khan , and K. Wang . 2018 . Towards Automatic Tuning of Apache Spark Configuration. In IEEE International Conference on Cloud Computing. 417--425 . N. Nguyen, M. Khan, and K. Wang. 2018. Towards Automatic Tuning of Apache Spark Configuration. In IEEE International Conference on Cloud Computing. 417--425."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"volume-title":"PETS: Bottleneck-Aware Spark Tuning with Parameter Ensembles. In International Conference on Computer Communication and Networks. 1--9.","author":"Perez T.","key":"e_1_3_2_1_31_1","unstructured":"T. Perez , W. Chen , R. Ji , L. Liu , and X. Zhou . 2018 . PETS: Bottleneck-Aware Spark Tuning with Parameter Ensembles. In International Conference on Computer Communication and Networks. 1--9. T. Perez, W. Chen, R. Ji, L. Liu, and X. Zhou. 2018. PETS: Bottleneck-Aware Spark Tuning with Parameter Ensembles. In International Conference on Computer Communication and Networks. 1--9."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2699414"},{"volume-title":"IEEE International Parallel and Distributed Processing Symposium Workshops. 1653--1659","author":"Sehrish S.","key":"e_1_3_2_1_33_1","unstructured":"S. Sehrish , J. Kowalkowski , and M. Paterno . 2016. Exploring the Performance of Spark for a Scientific Use Case . In IEEE International Parallel and Distributed Processing Symposium Workshops. 1653--1659 . S. Sehrish, J. Kowalkowski, and M. Paterno. 2016. Exploring the Performance of Spark for a Scientific Use Case. In IEEE International Parallel and Distributed Processing Symposium Workshops. 1653--1659."},{"volume-title":"Spark and HPC for High Energy Physics Data Analyses. In IEEE International Parallel and Distributed Processing Symposium Workshops. 1048--1057","author":"Sehrish S.","key":"e_1_3_2_1_34_1","unstructured":"S. Sehrish , J. Kowalkowski , and M. Paterno . 2017 . Spark and HPC for High Energy Physics Data Analyses. In IEEE International Parallel and Distributed Processing Symposium Workshops. 1048--1057 . S. Sehrish, J. Kowalkowski, and M. Paterno. 2017. Spark and HPC for High Energy Physics Data Analyses. In IEEE International Parallel and Distributed Processing Symposium Workshops. 1048--1057."},{"volume-title":"Spark Deployment and Performance Evaluation on the MareNostrum Supercomputer. In IEEE International Conference on Big Data. 299--306","author":"Tous R.","key":"e_1_3_2_1_35_1","unstructured":"R. Tous , A. Gounaris , C. Tripiana , J. Torres , S. Girona , E. Ayguad\u00e9 , J. Labarta , Y. Becerra , D. Carrera , and M. Valero . 2015 . Spark Deployment and Performance Evaluation on the MareNostrum Supercomputer. In IEEE International Conference on Big Data. 299--306 . R. Tous, A. Gounaris, C. Tripiana, J. Torres, S. Girona, E. Ayguad\u00e9, J. Labarta, Y. Becerra, D. Carrera, and M. Valero. 2015. Spark Deployment and Performance Evaluation on the MareNostrum Supercomputer. In IEEE International Conference on Big Data. 299--306."},{"volume-title":"Performance Evaluation of Big Data Frameworks for Large-Scale Data Analytics. In IEEE International Conference on Big Data. 424--431","author":"Veiga J.","key":"e_1_3_2_1_36_1","unstructured":"J. Veiga , R. Exp\u00f3sito , X. Pardo , G. Taboada , and J. Tourifio . 2016 . Performance Evaluation of Big Data Frameworks for Large-Scale Data Analytics. In IEEE International Conference on Big Data. 424--431 . J. Veiga, R. Exp\u00f3sito, X. Pardo, G. Taboada, and J. Tourifio. 2016. Performance Evaluation of Big Data Frameworks for Large-Scale Data Analytics. In IEEE International Conference on Big Data. 424--431."},{"key":"e_1_3_2_1_37_1","article-title":"In Situ Data Analysis and I\/O Acceleration of FLASH Astrophysics Simulation on Leadership-Class System Using GLEAN. In SciDAC","author":"Vishwanath V.","year":"2011","unstructured":"V. Vishwanath , M. Hereld , M. Papka , R. Hudson , G. Jordan , and C. Daley . 2011 . In Situ Data Analysis and I\/O Acceleration of FLASH Astrophysics Simulation on Leadership-Class System Using GLEAN. In SciDAC , Journal of Physics: Conference Series. V. Vishwanath, M. Hereld, M. Papka, R. Hudson, G. Jordan, and C. Daley. 2011. In Situ Data Analysis and I\/O Acceleration of FLASH Astrophysics Simulation on Leadership-Class System Using GLEAN. In SciDAC, Journal of Physics: Conference Series.","journal-title":"Journal of Physics: Conference Series."},{"volume-title":"Characterization and Optimization of Memory-Resident MapReduce on HPC Systems. In IEEE International Parallel and Distributed Processing Symposium. 799--808","author":"Wang Y.","key":"e_1_3_2_1_38_1","unstructured":"Y. Wang , R. Goldstone , W. Yu , and T. Wang . 2014 . Characterization and Optimization of Memory-Resident MapReduce on HPC Systems. In IEEE International Parallel and Distributed Processing Symposium. 799--808 . Y. Wang, R. Goldstone, W. Yu, and T. Wang. 2014. Characterization and Optimization of Memory-Resident MapReduce on HPC Systems. In IEEE International Parallel and Distributed Processing Symposium. 799--808."},{"volume-title":"Big Data Analytics on HPC Architectures: Performance and Cost. In IEEE International Conference on Big Data. 2286--2295","author":"Xenopoulos P.","key":"e_1_3_2_1_39_1","unstructured":"P. Xenopoulos , J. Daniel , M. Matheson , and S. Sukumar . 2016 . Big Data Analytics on HPC Architectures: Performance and Cost. In IEEE International Conference on Big Data. 2286--2295 . P. Xenopoulos, J. Daniel, M. Matheson, and S. Sukumar. 2016. Big Data Analytics on HPC Architectures: Performance and Cost. In IEEE International Conference on Big Data. 2286--2295."},{"key":"e_1_3_2_1_40_1","unstructured":"J. Zhang. 2018. HTuning Spark Performance. https:\/\/qspace.library.queensu.ca\/handle\/1974\/24439  J. Zhang. 2018. HTuning Spark Performance. https:\/\/qspace.library.queensu.ca\/handle\/1974\/24439"}],"event":{"name":"HPCAsia2020: International Conference on High Performance Computing in Asia-Pacific Region","sponsor":["IPSJ","SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing"],"location":"Fukuoka Japan","acronym":"HPCAsia2020"},"container-title":["Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368474.3368482","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368474.3368482","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:25Z","timestamp":1750197685000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368474.3368482"}},"subtitle":["Taming the Ephemeral Data Explosion"],"short-title":[],"issued":{"date-parts":[[2020,1,15]]},"references-count":40,"alternative-id":["10.1145\/3368474.3368482","10.1145\/3368474"],"URL":"https:\/\/doi.org\/10.1145\/3368474.3368482","relation":{},"subject":[],"published":{"date-parts":[[2020,1,15]]},"assertion":[{"value":"2020-01-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}