{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T01:10:15Z","timestamp":1788570615289,"version":"build-2803163510"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2016,9]]},"abstract":"<jats:p>The rising need for custom machine learning (ML) algorithms and the growing data sizes that require the exploitation of distributed, data-parallel frameworks such as MapReduce or Spark, pose significant productivity challenges to data scientists. Apache SystemML addresses these challenges through declarative ML by (1) increasing the productivity of data scientists as they are able to express custom algorithms in a familiar domain-specific language covering linear algebra primitives and statistical functions, and (2) transparently running these ML algorithms on distributed, data-parallel frameworks by applying cost-based compilation techniques to generate efficient, low-level execution plans with in-memory single-node and large-scale distributed operations. This paper describes SystemML on Apache Spark, end to end, including insights into various optimizer and runtime techniques as well as performance characteristics. We also share lessons learned from porting SystemML to Spark and declarative ML in general. Finally, SystemML is open-source, which allows the database community to leverage it as a testbed for further research.<\/jats:p>","DOI":"10.14778\/3007263.3007279","type":"journal-article","created":{"date-parts":[[2016,11,1]],"date-time":"2016-11-01T09:47:47Z","timestamp":1477993667000},"page":"1425-1436","source":"Crossref","is-referenced-by-count":144,"title":["SystemML"],"prefix":"10.14778","volume":"9","author":[{"given":"Matthias","family":"Boehm","sequence":"first","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael W.","family":"Dusenberry","sequence":"additional","affiliation":[{"name":"IBM Spark Technology Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deron","family":"Eriksson","sequence":"additional","affiliation":[{"name":"IBM Spark Technology Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandre V.","family":"Evfimievski","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faraz Makari","family":"Manshadi","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niketan","family":"Pansare","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Berthold","family":"Reinwald","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frederick R.","family":"Reiss","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden and IBM Spark Technology Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prithviraj","family":"Sen","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arvind C.","family":"Surve","sequence":"additional","affiliation":[{"name":"IBM Spark Technology Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shirish","family":"Tatikonda","sequence":"additional","affiliation":[{"name":"IBM Research --- Almaden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2016,9]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR, abs\/1603.04467","author":"Abadi M.","year":"2016","unstructured":"M. Abadi TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR, abs\/1603.04467 , 2016 . M. Abadi et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR, abs\/1603.04467, 2016."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0357-y"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750543"},{"key":"e_1_2_1_4_1","unstructured":"Apache. Mahout 2016. http:\/\/mahout.apache.org.  Apache. Mahout 2016. http:\/\/mahout.apache.org."},{"key":"e_1_2_1_5_1","unstructured":"Apache. Spark 2016. http:\/\/spark.apache.org.  Apache. Spark 2016. http:\/\/spark.apache.org."},{"key":"e_1_2_1_6_1","unstructured":"Apache. SystemML (incubating) 2016. http:\/\/systemml.apache.org.  Apache. SystemML (incubating) 2016. http:\/\/systemml.apache.org."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2688500.2688521"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732286.2732292"},{"key":"e_1_2_1_9_1","volume-title":"SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs","author":"Boehm M.","year":"2014","unstructured":"M. Boehm SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs . IEEE Data Eng. Bull ., 37(3), 2014 . M. Boehm et al. SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs. IEEE Data Eng. Bull., 37(3), 2014."},{"key":"e_1_2_1_10_1","volume-title":"Declarative Machine Learning --- A Classification of Basic Properties and Types. CoRR, abs\/1605.05826","author":"Boehm M.","year":"2016","unstructured":"M. Boehm Declarative Machine Learning --- A Classification of Basic Properties and Types. CoRR, abs\/1605.05826 , 2016 . M. Boehm et al. Declarative Machine Learning --- A Classification of Basic Properties and Types. CoRR, abs\/1605.05826, 2016."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465283"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824045"},{"key":"e_1_2_1_13_1","volume-title":"MLPACK: A Scalable C++ Machine Learning Library. JMLR, 14(1)","author":"Curtin R. R.","year":"2013","unstructured":"R. R. Curtin MLPACK: A Scalable C++ Machine Learning Library. JMLR, 14(1) , 2013 . R. R. Curtin et al. MLPACK: A Scalable C++ Machine Learning Library. JMLR, 14(1), 2013."},{"key":"e_1_2_1_14_1","volume-title":"OSDI","author":"Dean J.","year":"2004","unstructured":"J. Dean and S. Ghemawat . MapReduce: Simplified Data Processing on Large Clusters . In OSDI , 2004 . J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, 2004."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767930"},{"key":"e_1_2_1_16_1","unstructured":"H2O. H2O. http:\/\/h2o.ai\/product\/.  H2O. H2O . http:\/\/h2o.ai\/product\/."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367510"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465273"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749432"},{"key":"e_1_2_1_20_1","volume-title":"CIDR","author":"Kraska T.","year":"2013","unstructured":"T. Kraska : A Distributed Machine-learning System . In CIDR , 2013 . T. Kraska et al. MLbase: A Distributed Machine-learning System. In CIDR, 2013."},{"key":"e_1_2_1_21_1","volume-title":"Vowpal Wabbit Online Learning Project","author":"Langford J.","year":"2007","unstructured":"J. Langford , L. Li , and A. Strehl . Vowpal Wabbit Online Learning Project , 2007 . J. Langford, L. Li, and A. Strehl. Vowpal Wabbit Online Learning Project, 2007."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772760"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/2212351.2212354"},{"key":"e_1_2_1_24_1","volume-title":"Apache","author":"Lyubimov D.","year":"2016","unstructured":"D. Lyubimov . Mahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines . Apache , 2016 . D. Lyubimov. Mahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines. Apache, 2016."},{"key":"e_1_2_1_25_1","volume-title":"MLlib: Machine Learning in Apache Spark. CoRR, abs\/1505.06807","author":"Meng X.","year":"2015","unstructured":"X. Meng MLlib: Machine Learning in Apache Spark. CoRR, abs\/1505.06807 , 2015 . X. Meng et al. MLlib: Machine Learning in Apache Spark. CoRR, abs\/1505.06807, 2015."},{"key":"e_1_2_1_26_1","unstructured":"Microsoft. Distributed Machine Learning Toolkit.  Microsoft. Distributed Machine Learning Toolkit ."},{"key":"e_1_2_1_27_1","volume-title":"CRAN Task View: High-Performance and Parallel Computing with R","year":"2016","unstructured":"R-project. CRAN Task View: High-Performance and Parallel Computing with R , 2016 . R-project. CRAN Task View: High-Performance and Parallel Computing with R, 2016."},{"key":"e_1_2_1_28_1","volume-title":"Revolution R Enterprise ScaleR","year":"2016","unstructured":"Revolution Analytics. Revolution R Enterprise ScaleR , 2016 . Revolution Analytics. Revolution R Enterprise ScaleR, 2016."},{"key":"e_1_2_1_29_1","unstructured":"Skytree. Skytree Machine Learning Software.  Skytree. Skytree Machine Learning Software ."},{"key":"e_1_2_1_30_1","first-page":"11","year":"2010","unstructured":"Sonnenburg The SHOGUN Machine Learning Toolbox. JMLR , 11 , 2010 . Sonnenburg et al. The SHOGUN Machine Learning Toolbox. JMLR, 11, 2010.","journal-title":"The SHOGUN Machine Learning Toolbox. JMLR"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2013.158"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806777.2806945"},{"key":"e_1_2_1_33_1","volume-title":"SSDBM","author":"Stonebraker M.","year":"2011","unstructured":"M. Stonebraker The Architecture of SciDB . In SSDBM , 2011 . M. Stonebraker et al. The Architecture of SciDB. In SSDBM, 2011."},{"key":"e_1_2_1_34_1","volume-title":"ICML","author":"Sujeeth A. K.","year":"2011","unstructured":"A. K. Sujeeth : An Implicitly Parallel Domain-Specific Language for Machine Learning . In ICML , 2011 . A. K. Sujeeth et al. OptiML: An Implicitly Parallel Domain-Specific Language for Machine Learning. In ICML, 2011."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.12"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465371"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742793"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723712"},{"key":"e_1_2_1_40_1","volume-title":"linalg: Matrix Computations in Apache Spark. CoRR, abs\/1509.02256","author":"Zadeh R. B.","year":"2015","unstructured":"R. B. Zadeh linalg: Matrix Computations in Apache Spark. CoRR, abs\/1509.02256 , 2015 . R. B. Zadeh et al. linalg: Matrix Computations in Apache Spark. CoRR, abs\/1509.02256, 2015."},{"key":"e_1_2_1_41_1","volume-title":"NSDI","author":"Zaharia M.","year":"2012","unstructured":"M. Zaharia Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing . In NSDI , 2012 . M. Zaharia et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In NSDI, 2012."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2593678"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-68880-8_32"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3007263.3007279","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:38:17Z","timestamp":1672202297000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3007263.3007279"}},"subtitle":["declarative machine learning on spark"],"short-title":[],"issued":{"date-parts":[[2016,9]]},"references-count":43,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2016,9]]}},"alternative-id":["10.14778\/3007263.3007279"],"URL":"https:\/\/doi.org\/10.14778\/3007263.3007279","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2016,9]]}}}