{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:56:07Z","timestamp":1772909767169,"version":"3.50.1"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"8","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2017,4]]},"abstract":"<jats:p>Apache Spark is a popular framework for data analytics with attractive features such as fault tolerance and interoperability with the Hadoop ecosystem. Unfortunately, many analytics operations in Spark are an order of magnitude or more slower compared to native implementations written with high performance computing tools such as MPI. There is a need to bridge the performance gap while retaining the benefits of the Spark ecosystem such as availability, productivity, and fault tolerance. In this paper, we propose a system for integrating MPI with Spark and analyze the costs and benefits of doing so for four distributed graph and machine learning applications. We show that offloading computation to an MPI environment from within Spark provides 3.1\u221217.7\u00d7 speedups on the four sparse applications, including all of the overheads. This opens up an avenue to reuse existing MPI libraries in Spark with little effort.<\/jats:p>","DOI":"10.14778\/3090163.3090168","type":"journal-article","created":{"date-parts":[[2017,5,12]],"date-time":"2017-05-12T12:17:14Z","timestamp":1494591434000},"page":"901-912","source":"Crossref","is-referenced-by-count":42,"title":["Bridging the gap between HPC and big data frameworks"],"prefix":"10.14778","volume":"10","author":[{"given":"Michael","family":"Anderson","sequence":"first","affiliation":[{"name":"Parallel Computing Lab"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaden","family":"Smith","sequence":"additional","affiliation":[{"name":"University of Minnesota"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Narayanan","family":"Sundaram","sequence":"additional","affiliation":[{"name":"Parallel Computing Lab"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mihai","family":"Capot\u0103","sequence":"additional","affiliation":[{"name":"Parallel Computing Lab"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zheguang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Brown University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Subramanya","family":"Dulloor","sequence":"additional","affiliation":[{"name":"Intel Corporation"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nadathur","family":"Satish","sequence":"additional","affiliation":[{"name":"Parallel Computing Lab"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Theodore L.","family":"Willke","sequence":"additional","affiliation":[{"name":"Parallel Computing Lab"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806534"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2016.86"},{"key":"e_1_2_1_3_1","first-page":"27","volume-title":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2009","author":"Asuncion A.","year":"2009"},{"key":"e_1_2_1_4_1","unstructured":"M. Axtmann T. Bingmann E. J\u00f6bstl S. Lamm H. C. Nguyen A. Noe M. Stumpp P. Sanders S. Schlag and T. Sturm. Thrill - distributed big data batch processing framework in C++. http:\/\/project-thrill.org\/ 2016.  M. Axtmann T. Bingmann E. J\u00f6bstl S. Lamm H. C. Nguyen A. Noe M. Stumpp P. Sanders S. Schlag and T. Sturm. Thrill - distributed big data batch processing framework in C++. http:\/\/project-thrill.org\/ 2016."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944937"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939675"},{"key":"e_1_2_1_7_1","unstructured":"deeplearning4j. Deep Learning for Java. Open-Source Distributed Deep Learning Library for the JVM. http:\/\/deeplearning4j.org\/ 2016.  deeplearning4j. Deep Learning for Java. Open-Source Distributed Deep Learning Library for the JVM. http:\/\/deeplearning4j.org\/ 2016."},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/3-540-45255-9_47","volume-title":"Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM\/MPI)","author":"Fagg G. E.","year":"2000"},{"key":"e_1_2_1_9_1","unstructured":"M. P. I. Forum. Mpi: A message-passing interface standard version 3.1. Technical report 2015.  M. P. I. Forum. Mpi: A message-passing interface standard version 3.1. Technical report 2015."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2014.78"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840606"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2907294.2907307"},{"key":"e_1_2_1_13_1","unstructured":"H2O.ai. Sparkling Water. https:\/\/github.com\/h2oai\/sparkling-water 2016.  H2O.ai. Sparkling Water. https:\/\/github.com\/h2oai\/sparkling-water 2016."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.Congress.2014.137"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807624"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of Link Analysis, Counterterrorism and Security 2006","author":"Kolda T. G.","year":"2006"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1137\/07070111X"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772751"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2014.90"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507163"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946679"},{"key":"e_1_2_1_22_1","first-page":"293","volume-title":"Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation, NSDI 2015","author":"Ousterhout K.","year":"2015"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2011.240"},{"issue":"121","key":"e_1_2_1_24_1","first-page":"130","article-title":"Big data analytics in the cloud: Spark on Hadoop vs MPI\/OpenMP on Beowulf","volume":"53","author":"Reyes-Ortiz J. L.","year":"2015","journal-title":"Procedia Computer Science"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610518"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348308"},{"key":"e_1_2_1_27_1","unstructured":"N. D. Sidiropoulos L. De Lathauwer X. Fu K. Huang E. E. Papalexakis and C. Faloutsos. Tensor decomposition for signal processing and machine learning. arXiv preprint arXiv:1607.01668 2016.  N. D. Sidiropoulos L. De Lathauwer X. Fu K. Huang E. E. Papalexakis and C. Faloutsos. Tensor decomposition for signal processing and machine learning. arXiv preprint arXiv:1607.01668 2016."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2016.93"},{"key":"e_1_2_1_29_1","unstructured":"S. Smith J. W. Choi J. Li R. Vuduc J. Park X. Liu and G. Karypis. FROSTT: The formidable repository of open sparse tensors and tools 2017.  S. Smith J. W. Choi J. Li R. Vuduc J. Park X. Liu and G. Karypis. FROSTT: The formidable repository of open sparse tensors and tools 2017."},{"key":"e_1_2_1_30_1","unstructured":"S. Smith and G. Karypis. SPLATT: The Surprisingly ParalleL spArse Tensor Toolkit. http:\/\/cs.umn.edu\/~splatt\/.  S. Smith and G. Karypis. SPLATT: The Surprisingly ParalleL spArse Tensor Toolkit. http:\/\/cs.umn.edu\/~splatt\/."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2016.113"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2809974.2809983"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484425.2484427"},{"key":"e_1_2_1_34_1","first-page":"2","volume-title":"Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI 2012","author":"Zaharia M.","year":"2012"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3090163.3090168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:20:03Z","timestamp":1672222803000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3090163.3090168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,4]]},"references-count":34,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2017,4]]}},"alternative-id":["10.14778\/3090163.3090168"],"URL":"https:\/\/doi.org\/10.14778\/3090163.3090168","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2017,4]]}}}