{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T05:21:52Z","timestamp":1672291312853},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2016,9]]},"abstract":"<jats:p>Due to R's popularity as a data-mining tool, many distributed systems expose an R-based API to users who need to build a distributed application in R. As a result, data scientists have to learn to use different interfaces such as RHadoop, SparkR, Revolution R's ScaleR, and HPE's Distributed R. Unfortunately, these interfaces are custom, non-standard, and difficult to learn. Not surprisingly, R applications written in one framework do not work in another, and each backend infrastructure has spent redundant effort in implementing distributed machine learning algorithms.<\/jats:p>\n          <jats:p>Working with the members of R-core, we have created ddR (Distributed Data structures in R), a unified system that works across different distributed frameworks. In ddR, we introduce a novel programming primitive called dmapply that executes functions on distributed data structures. The dmapply primitive encapsulates different computation patterns: from function and data broadcast to pair-wise communication. We show that dmapply is powerful enough to express algorithms that fit the statistical query model, which includes many popular machine learning algorithms, as well as applications written in MapReduce. We have integrated ddR with many backends, such as R's single-node parallel framework, multi-node SNOW framework, Spark, and HPE Distributed R, with few or no modifications to any of these systems. We have also implemented multiple machine learning algorithms which are not only portable across different distributed systems, but also have performance comparable to the \"native\" implementations on the backends. We believe that ddR will standardize distributed computing in R, just like the SQL interface has standardized how relational data is manipulated.<\/jats:p>","DOI":"10.14778\/3007263.3007268","type":"journal-article","created":{"date-parts":[[2016,11,1]],"date-time":"2016-11-01T13:47:47Z","timestamp":1478008067000},"page":"1293-1304","source":"Crossref","is-referenced-by-count":1,"title":["dmapply"],"prefix":"10.14778","volume":"9","author":[{"given":"Edward","family":"Ma","sequence":"first","affiliation":[{"name":"HPE Vertica"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vishrut","family":"Gupta","sequence":"additional","affiliation":[{"name":"HPE Vertica"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Meichun","family":"Hsu","sequence":"additional","affiliation":[{"name":"HPE Vertica"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Indrajit","family":"Roy","sequence":"additional","affiliation":[{"name":"Hewlett Packard Labs"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,9]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Apache Mahout. http:\/\/mahout.apache.org.  Apache Mahout. http:\/\/mahout.apache.org."},{"key":"e_1_2_1_2_1","unstructured":"Comparing pattern mining on a billion records with HP Vertica and Hadoop. http:\/\/www.vertica.com\/2013\/04\/08\/comparing-pattern-mining-on-a-billion-records-with-hp-vertica-and-hadoop\/.  Comparing pattern mining on a billion records with HP Vertica and Hadoop. http:\/\/www.vertica.com\/2013\/04\/08\/comparing-pattern-mining-on-a-billion-records-with-hp-vertica-and-hadoop\/."},{"key":"e_1_2_1_3_1","unstructured":"H2O: Machine learning library. http:\/\/www.h2o.ai\/.  H2O: Machine learning library. http:\/\/www.h2o.ai\/."},{"key":"e_1_2_1_4_1","unstructured":"MATLAB---parallel computing toolbox. http:\/\/www.mathworks.com\/products\/parallel-computing\/.  MATLAB---parallel computing toolbox. http:\/\/www.mathworks.com\/products\/parallel-computing\/."},{"key":"e_1_2_1_5_1","unstructured":"Revolution R enterprise scaler. http:\/\/www.revolutionanalytics.com\/revolution-r-enterprise-scaler.  Revolution R enterprise scaler. http:\/\/www.revolutionanalytics.com\/revolution-r-enterprise-scaler."},{"key":"e_1_2_1_6_1","unstructured":"RHadoop and MapR. https:\/\/www.mapr.com\/resources\/rhadoop-and-mapr.  RHadoop and MapR. https:\/\/www.mapr.com\/resources\/rhadoop-and-mapr."},{"key":"e_1_2_1_7_1","unstructured":"The s4 object system. http:\/\/adv-r.had.co.nz\/S4.html.  The s4 object system. http:\/\/adv-r.had.co.nz\/S4.html."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732286.2732292"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807275"},{"key":"e_1_2_1_10_1","first-page":"10","volume-title":"Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation -","volume":"6","author":"Dean J.","year":"2004","unstructured":"J. Dean and S. Ghemawat . Mapreduce: Simplified data processing on large clusters . In Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation - Volume 6 , OSDI'04, pages 10 -- 10 , Berkeley, CA, USA , 2004 . USENIX Association. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation - Volume 6, OSDI'04, pages 10--10, Berkeley, CA, USA, 2004. USENIX Association."},{"key":"e_1_2_1_11_1","unstructured":"D. Eddelbuettel. High-performance and parallel computing with r. https:\/\/cran.r-project.org\/web\/views\/HighPerformanceComputing.html.  D. Eddelbuettel. High-performance and parallel computing with r. https:\/\/cran.r-project.org\/web\/views\/HighPerformanceComputing.html."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767930"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1137\/0613024"},{"key":"e_1_2_1_14_1","volume-title":"Bridging two worlds with RICE integrating R into the SAP in-memory computing engine. PVLDB, 4(12)","author":"Gro\u00dfe P.","year":"2011","unstructured":"P. Gro\u00dfe , W. Lehner , T. Weichert , F. F\u00e4rber , and W.-S. Li . Bridging two worlds with RICE integrating R into the SAP in-memory computing engine. PVLDB, 4(12) , 2011 . P. Gro\u00dfe, W. Lehner, T. Weichert, F. F\u00e4rber, and W.-S. Li. Bridging two worlds with RICE integrating R into the SAP in-memory computing engine. PVLDB, 4(12), 2011."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367510"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/293347.293351"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/355841.355847"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45065-5_15"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/2212351.2212354"},{"key":"e_1_2_1_20_1","unstructured":"E. Ma I. Roy and M. Lawrence. ddR: Distributed data structures in R. https:\/\/cran.r-project.org\/web\/packages\/ddR\/index.html.  E. Ma I. Roy and M. Lawrence. ddR: Distributed data structures in R. https:\/\/cran.r-project.org\/web\/packages\/ddR\/index.html."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807184"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522738"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376726"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742789"},{"key":"e_1_2_1_25_1","unstructured":"D. Robinson. Package downloads over time. https:\/\/dgrtwo.shinyapps.io\/cranview\/.  D. Robinson. Package downloads over time. https:\/\/dgrtwo.shinyapps.io\/cranview\/."},{"key":"e_1_2_1_26_1","unstructured":"D. Smith. New surveys show continued popularity of R. http:\/\/blog.revolutionanalytics.com\/2015\/11\/new-surveys-show-continued-popularity-of-r.html.  D. Smith. New surveys show continued popularity of R. http:\/\/blog.revolutionanalytics.com\/2015\/11\/new-surveys-show-continued-popularity-of-r.html."},{"key":"e_1_2_1_27_1","first-page":"281","volume-title":"S. K. Kim, Y. an Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. Map-reduce for machine learning on multicore","author":"C.","year":"2007","unstructured":"C. tao Chu , S. K. Kim, Y. an Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. Map-reduce for machine learning on multicore . In B. Sch\u00f6lkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 281 -- 288 . MIT Press , 2007 . C. tao Chu, S. K. Kim, Y. an Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. Map-reduce for machine learning on multicore. In B. Sch\u00f6lkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 281--288. MIT Press, 2007."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595641"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465371"},{"key":"e_1_2_1_31_1","volume-title":"4th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud'12","author":"Venkataraman S.","year":"2012","unstructured":"S. Venkataraman , I. Roy , A. AuYoung , and R. S. Schreiber . Using R for iterative and incremental processing . In 4th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud'12 , Boston, MA, USA , June 12-13, 2012 , 2012. S. Venkataraman, I. Roy, A. AuYoung, and R. S. Schreiber. Using R for iterative and incremental processing. In 4th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud'12, Boston, MA, USA, June 12-13, 2012, 2012."},{"key":"e_1_2_1_32_1","unstructured":"H. Wickham. dplyr:A grammar for data manipulation. https:\/\/cran.r-project.org\/web\/packages\/dplyr\/index.html.  H. Wickham. dplyr:A grammar for data manipulation. https:\/\/cran.r-project.org\/web\/packages\/dplyr\/index.html."},{"key":"e_1_2_1_33_1","first-page":"1","volume-title":"Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08","author":"Yu Y.","year":"2008","unstructured":"Y. Yu , M. Isard , D. Fetterly , M. Budiu , U. Erlingsson , P. K. Gunda , and J. Currey . Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language . In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08 , pages 1 -- 14 , Berkeley, CA, USA , 2008 . USENIX Association. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08, pages 1--14, Berkeley, CA, USA, 2008. USENIX Association."},{"key":"e_1_2_1_34_1","first-page":"2","volume-title":"Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI'12","author":"Zaharia M.","year":"2012","unstructured":"M. Zaharia , M. Chowdhury , T. Das , A. Dave , J. Ma , M. McCauley , M. J. Franklin , S. Shenker , and I. Stoica . Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing . In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI'12 , pages 2 -- 2 , Berkeley, CA, USA , 2012 . USENIX Association. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI'12, pages 2--2, Berkeley, CA, USA, 2012. USENIX Association."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522737"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3007263.3007268","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T09:34:32Z","timestamp":1672220072000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3007263.3007268"}},"subtitle":["a functional primitive to express distributed machine learning algorithms in R"],"short-title":[],"issued":{"date-parts":[[2016,9]]},"references-count":35,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2016,9]]}},"alternative-id":["10.14778\/3007263.3007268"],"URL":"https:\/\/doi.org\/10.14778\/3007263.3007268","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2016,9]]}}}