{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T22:43:42Z","timestamp":1777675422444,"version":"3.51.4"},"reference-count":31,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2017,7,2]],"date-time":"2017-07-02T00:00:00Z","timestamp":1498953600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2018,1]]},"abstract":"<jats:p>With the ever-increasing need to analyze large amounts of data to get useful insights, it is essential to develop complex parallel machine learning algorithms that can scale with data and number of parallel processes. These algorithms need to run on large data sets as well as they need to be executed with minimal time in order to extract useful information in a time-constrained environment. Message passing interface (MPI) is a widely used model for developing such algorithms in high-performance computing paradigm, while Apache Spark and Apache Flink are emerging as big data platforms for large-scale parallel machine learning. Even though these big data frameworks are designed differently, they follow the data flow model for execution and user APIs. Data flow model offers fundamentally different capabilities than the MPI execution model, but the same type of parallelism can be used in applications developed in both models. This article presents three distinct machine learning algorithms implemented in MPI, Spark, and Flink and compares their performance and identifies strengths and weaknesses in each platform.<\/jats:p>","DOI":"10.1177\/1094342017712976","type":"journal-article","created":{"date-parts":[[2017,7,14]],"date-time":"2017-07-14T17:34:19Z","timestamp":1500053659000},"page":"61-73","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":25,"title":["Anatomy of machine learning algorithm implementations in MPI, Spark, and Flink"],"prefix":"10.1177","volume":"32","author":[{"given":"Supun","family":"Kamburugamuve","sequence":"first","affiliation":[{"name":"School of Informatics and Computing Indiana University, Bloomington, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pulasthi","family":"Wickramasinghe","sequence":"additional","affiliation":[{"name":"School of Informatics and Computing Indiana University, Bloomington, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saliya","family":"Ekanayake","sequence":"additional","affiliation":[{"name":"Network Dynamics and Simulation Science Laboratory Biocomplexity Institute, Virginia Tech, Blacksburg, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Geoffrey C","family":"Fox","sequence":"additional","affiliation":[{"name":"School of Informatics and Computing Indiana University, Bloomington, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2017,7,2]]},"reference":[{"key":"bibr1-1094342017712976","first-page":"265","volume-title":"12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)","author":"Abadi M","year":"2016"},{"key":"bibr2-1094342017712976","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"bibr3-1094342017712976","unstructured":"Apache Giraph (n.d.) Available at: http:\/\/giraph.apache.org\/ (accessed 14 December 2016)."},{"key":"bibr4-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2010.45"},{"issue":"4","key":"bibr5-1094342017712976","volume":"36","author":"Carbone P","year":"2015","journal-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering"},{"key":"bibr6-1094342017712976","unstructured":"Doekemeijer N, Varbanescu AL (2014) A survey of parallel graph processing frameworks, Delft, Netherlands: Delft University of Technology. Available at: http:\/\/www.ds.ewi.tudelft.nl\/fileadmin\/pds\/reports\/2014\/PDS-2014-003.pdf (accessed January 2016)."},{"key":"bibr7-1094342017712976","doi-asserted-by":"crossref","unstructured":"Ekanayake J, Li H, Zhang B, (2010) Twister: a runtime for iterative map reduce. In: Proceedings of the 19th ACM international symposium on high performance distributed computing\u2019, HPDC\u2019 10, pp. 810\u2013818. New York, NY, USA: ACM. Available at: http:\/\/doi.acm.org\/10.1145\/1851476.1851593","DOI":"10.1145\/1851476.1851593"},{"key":"bibr8-1094342017712976","doi-asserted-by":"crossref","unstructured":"Ekanayake S, Kamburugamuve S, Fox GC (2016) Spidal java: high performance data analytics with java and mpi on large multicore hpc clusters. In: Proceedings of the 24th high performance computing symposium\u2019, HPC \u201916, San Diego, CA, USA, pp. 3:1\u20133:8. Available at: http:\/\/dx.doi.org\/10.22360\/SpringSim.2016.HPC.031","DOI":"10.22360\/SpringSim.2016.HPC.031"},{"key":"bibr9-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840622"},{"key":"bibr10-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49748-8_1"},{"key":"bibr11-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2015.122"},{"key":"bibr12-1094342017712976","first-page":"599","volume-title":"11th USENIX symposium on operating systems design and implementation (OSDI 14)","author":"Gonzalez JE","year":"2014"},{"key":"bibr13-1094342017712976","unstructured":"H2O (2017) Available at: http:\/\/www.h2o.ai\/h2o\/ (accessed 02 January 2017)."},{"key":"bibr14-1094342017712976","unstructured":"Intel Data Analytics Acceleration Library (n.d.) Available at: https:\/\/software.intel.com\/en-us\/intel-daal (accessed 02 January 2017)."},{"key":"bibr15-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1007\/BF02289565"},{"key":"bibr16-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-13021-7_9"},{"key":"bibr17-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2013.78"},{"key":"bibr18-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2014.90"},{"key":"bibr19-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807184"},{"key":"bibr20-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2016.22"},{"issue":"34","key":"bibr21-1094342017712976","first-page":"1","volume":"17","author":"Meng X","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"bibr22-1094342017712976","unstructured":"Owen S, Anil R, Dunning T, (2011) Mahout in action, 1st edn. Shelter Island, NY, USA: Manning."},{"key":"bibr23-1094342017712976","unstructured":"Project Tungsten: Bringing Apache Spark Closer to Bare Metal (2015) https:\/\/databricks.com\/blog\/2015\/04\/28\/project-tungsten-bringing-spark-closer-to-bare-metal.html"},{"key":"bibr24-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.07.286"},{"key":"bibr25-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2013.30"},{"key":"bibr26-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742790"},{"key":"bibr27-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2010.17"},{"key":"bibr28-1094342017712976","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2014.6835958"},{"key":"bibr29-1094342017712976","unstructured":"White T (2009) Hadoop: the definitive guide, 1st edn. Sebastopol, CA, USA: O\u2019Reilly Media, Inc."},{"key":"bibr30-1094342017712976","first-page":"2","volume-title":"Proceedings of the 9th USENIX conference on networked systems design and implementation\u2019, NSDI\u201912","author":"Zaharia M","year":"2012"},{"key":"bibr31-1094342017712976","unstructured":"Zaharia M, Chowdhury M, Franklin MJ, (2010) Spark: cluster computing with working sets. In: Proceedings of the 2Nd USENIX conference on hot topics in cloud computing, HotCloud\u201910, USENIX Association, Berkeley, CA, USA, pp. 10\u201310. Available at: http:\/\/dl.acm.org\/citation.cfm?id=1863103.1863113."}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017712976","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342017712976","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017712976","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:15:30Z","timestamp":1777450530000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342017712976"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,2]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,1]]}},"alternative-id":["10.1177\/1094342017712976"],"URL":"https:\/\/doi.org\/10.1177\/1094342017712976","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,7,2]]}}}