{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,20]],"date-time":"2026-09-20T03:34:46Z","timestamp":1789875286081,"version":"4.0.1"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"9","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2018,5]]},"abstract":"<jats:p>\n                    Modern analytics applications use a diverse mix of libraries and functions. Unfortunately, there is no optimization across these libraries, resulting in performance penalties as high as an order of magnitude in many applications. To address this problem, we proposed Weld, a\n                    <jats:italic>common runtime<\/jats:italic>\n                    for existing data analytics libraries that performs key physical optimizations such as pipelining under existing, imperative library APIs. In this work, we further develop the Weld vision by designing an automatic adaptive optimizer for Weld applications, and evaluating its impact on realistic data science workloads. Our optimizer eliminates multiple forms of overhead that arise when composing imperative libraries like Pandas and NumPy, and uses lightweight measurements to make data-dependent decisions at run-time in ad-hoc workloads where no statistics are available, with sub-second overhead. We also evaluate which optimizations have the largest impact in practice and whether Weld can be integrated into libraries incrementally. Our results are promising: using our optimizer, Weld accelerates data science workloads by up to 23X on one thread and 80X on eight threads, and its adaptive optimizations provide up to a 3.75X speedup over rule-based optimization. Moreover, Weld provides benefits if even just 4--5 operators in a library are ported to use it. Our results show that common runtime designs like Weld may be a viable approach to accelerate analytics.\n                  <\/jats:p>","DOI":"10.14778\/3213880.3213890","type":"journal-article","created":{"date-parts":[[2018,6,12]],"date-time":"2018-06-12T14:15:11Z","timestamp":1528812911000},"page":"1002-1015","source":"Crossref","is-referenced-by-count":55,"title":["Evaluating end-to-end optimization for data analytics applications in weld"],"prefix":"10.14778","volume":"11","author":[{"given":"Shoumik","family":"Palkar","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Thomas","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepak","family":"Narayanan","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pratiksha","family":"Thaker","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rahul","family":"Palamuttam","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Parimajan","family":"Negi","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anil","family":"Shanbhag","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Malte","family":"Schwarzkopf","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Holger","family":"Pirk","sequence":"additional","affiliation":[{"name":"Imperial College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saman","family":"Amarasinghe","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samuel","family":"Madden","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matei","family":"Zaharia","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,5]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"265","volume-title":"TensorFlow: A System for Large-Scale Machine Learning. In Proc. USENIX OSDI","author":"Abadi M.","year":"2016","unstructured":"M. Abadi , P. Barham , J. Chen , Z. Chen , A. Davis , J. Dean , M. Devin , S. Ghemawat , G. Irving , M. Isard , TensorFlow: A System for Large-Scale Machine Learning. In Proc. USENIX OSDI , pages 265 -- 283 , 2016 . M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. TensorFlow: A System for Large-Scale Machine Learning. In Proc. USENIX OSDI, pages 265--283, 2016."},{"key":"e_1_2_1_2_1","volume-title":"Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop. https:\/\/databricks.com\/blog\/2016\/05\/23\/","author":"Agarwal S.","year":"2016","unstructured":"S. Agarwal , D. Liu , and R. Xin . Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop. https:\/\/databricks.com\/blog\/2016\/05\/23\/ , 2016 . S. Agarwal, D. Liu, and R. Xin. Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop. https:\/\/databricks.com\/blog\/2016\/05\/23\/, 2016."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750543"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_5_1","volume-title":"https:\/\/arrow.apache.org\/","author":"Arrow Apache","year":"2018","unstructured":"Apache Arrow . https:\/\/arrow.apache.org\/ , 2018 . Apache Arrow. https:\/\/arrow.apache.org\/, 2018."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/173284.155343"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1006\/jpdc.1996.0107"},{"key":"e_1_2_1_8_1","volume-title":"http:\/\/bohrium.readthedocs.io","year":"2018","unstructured":"Bohrium. http:\/\/bohrium.readthedocs.io , 2018 . Bohrium. http:\/\/bohrium.readthedocs.io, 2018."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854038.2854042"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/181550.181564"},{"key":"e_1_2_1_11_1","volume-title":"https:\/\/cran.r-project.org","year":"2018","unstructured":"Cran. https:\/\/cran.r-project.org , 2018 . Cran. https:\/\/cran.r-project.org, 2018."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824045"},{"key":"e_1_2_1_13_1","volume-title":"http:\/\/www.nvidia.com\/object\/cuda_home_new.html","author":"CUDA.","year":"2018","unstructured":"CUDA. http:\/\/www.nvidia.com\/object\/cuda_home_new.html , 2018 . CUDA. http:\/\/www.nvidia.com\/object\/cuda_home_new.html, 2018."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687584"},{"key":"e_1_2_1_15_1","volume-title":"http:\/\/cython.org","year":"2018","unstructured":"Cython. http:\/\/cython.org , 2018 . Cython. http:\/\/cython.org, 2018."},{"key":"e_1_2_1_16_1","unstructured":"Pandas Cookbook Chapter 7: Cleaning Up Messy Data. https:\/\/github.com\/jvns\/pandas- cookbook\/.  Pandas Cookbook Chapter 7: Cleaning Up Messy Data. https:\/\/github.com\/jvns\/pandas- cookbook\/."},{"key":"e_1_2_1_17_1","volume-title":"https:\/\/en.wikipedia.org\/wiki\/Demand_paging","author":"Paging Demand","year":"2018","unstructured":"Demand Paging . https:\/\/en.wikipedia.org\/wiki\/Demand_paging , 2018 . Demand Paging. https:\/\/en.wikipedia.org\/wiki\/Demand_paging, 2018."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000001"},{"key":"e_1_2_1_19_1","unstructured":"Flight Delays and Cancellations Dataset. https:\/\/www.kaggle.com\/usdot\/flight-delays\/data.  Flight Delays and Cancellations Dataset. https:\/\/www.kaggle.com\/usdot\/flight-delays\/data."},{"key":"e_1_2_1_20_1","unstructured":"Gluon. https:\/\/gluon.mxnet.io.  Gluon. https:\/\/gluon.mxnet.io."},{"key":"e_1_2_1_21_1","volume-title":"Black Scholes Formula","author":"Goseme J.","year":"2013","unstructured":"J. Goseme . Black Scholes Formula , 2013 . J. Goseme. Black Scholes Formula, 2013."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/93605.98720"},{"key":"e_1_2_1_23_1","first-page":"288","volume-title":"A Versatile Representation for Queries","author":"Grust T.","year":"2004","unstructured":"T. Grust . Monad Comprehensions : A Versatile Representation for Queries , pages 288 -- 311 . Springer Berlin Heidelberg , Berlin, Heidelberg , 2004 . T. Grust. Monad Comprehensions: A Versatile Representation for Queries, pages 288--311. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004."},{"key":"e_1_2_1_24_1","unstructured":"J. Hamrick. The Demise of for Loops. https:\/\/docs.scipy.org\/doc\/numpy-1.13.0\/reference\/arrays.indexing.html.  J. Hamrick. The Demise of for Loops. https:\/\/docs.scipy.org\/doc\/numpy-1.13.0\/reference\/arrays.indexing.html."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_2_1_26_1","volume-title":"goo.gl\/dqwmrG","author":"Heisler S.","year":"2017","unstructured":"S. Heisler . A Beginner's Guide to Optimizing Pandas Code for Speed . goo.gl\/dqwmrG , 2017 . S. Heisler. A Beginner's Guide to Optimizing Pandas Code for Speed. goo.gl\/dqwmrG, 2017."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544927"},{"key":"e_1_2_1_28_1","volume-title":"http:\/\/hyper-db.de\/interface.html","author":"Web Interface HyPer","year":"2013","unstructured":"HyPer Web Interface . http:\/\/hyper-db.de\/interface.html , 2013 . HyPer Web Interface. http:\/\/hyper-db.de\/interface.html, 2013."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544810"},{"key":"e_1_2_1_30_1","first-page":"301","volume-title":"International Workshop on Languages and Compilers for Parallel Computing","author":"Kennedy K.","year":"1993","unstructured":"K. Kennedy and K. S. McKinley . Maximizing loop parallelism and improving data locality via loop fusion and distribution . In International Workshop on Languages and Compilers for Parallel Computing , pages 301 -- 320 . Springer , 1993 . K. Kennedy and K. S. McKinley. Maximizing loop parallelism and improving data locality via loop fusion and distribution. In International Workshop on Languages and Compilers for Parallel Computing, pages 301--320. Springer, 1993."},{"key":"e_1_2_1_31_1","volume-title":"An introduction to SPIR-V. https:\/\/www.khronos.org\/registry\/spir-v\/papers\/WhitePaper.pdf","author":"Kessenich J.","year":"2015","unstructured":"J. Kessenich . An introduction to SPIR-V. https:\/\/www.khronos.org\/registry\/spir-v\/papers\/WhitePaper.pdf , 2015 . J. Kessenich. An introduction to SPIR-V. https:\/\/www.khronos.org\/registry\/spir-v\/papers\/WhitePaper.pdf, 2015."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732951.2732959"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/977395.977673"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/355841.355847"},{"key":"e_1_2_1_35_1","unstructured":"W. Liu. Python and Pandas Part 4: More Baby Names. http:\/\/beyondvalence.blogspot.com\/2014\/09\/python-and-pandas-part-4-more-baby-names.html 2014.  W. Liu. Python and Pandas Part 4: More Baby Names. http:\/\/beyondvalence.blogspot.com\/2014\/09\/python-and-pandas-part-4-more-baby-names.html 2014."},{"key":"e_1_2_1_36_1","volume-title":"https:\/\/www.cs.umd.edu\/class\/fall2001\/cmsc411\/proj01\/proja\/loop.html","author":"Unrolling Loop","year":"2001","unstructured":"Loop Unrolling . https:\/\/www.cs.umd.edu\/class\/fall2001\/cmsc411\/proj01\/proja\/loop.html , 2001 . Loop Unrolling. https:\/\/www.cs.umd.edu\/class\/fall2001\/cmsc411\/proj01\/proja\/loop.html, 2001."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/PACT.2011.68"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1287369.1287387"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"e_1_2_1_40_1","volume-title":"https:\/\/software.intel.com\/en-us\/mkl","author":"Kernel Library Intel Math","year":"2018","unstructured":"Intel Math Kernel Library . https:\/\/software.intel.com\/en-us\/mkl , 2018 . Intel Math Kernel Library. https:\/\/software.intel.com\/en-us\/mkl, 2018."},{"key":"e_1_2_1_41_1","unstructured":"MNIST. http:\/\/yann.lecun.com\/exdb\/mnist\/.  MNIST. http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/2002938.2002940"},{"key":"e_1_2_1_43_1","unstructured":"NumPy. http:\/\/www.numpy.org\/.  NumPy. http:\/\/www.numpy.org\/."},{"key":"e_1_2_1_44_1","volume-title":"https:\/\/docs.scipy.org\/doc\/numpy-1.13.0\/reference\/arrays.indexing.html","author":"Array Indexing NumPy","year":"2009","unstructured":"NumPy Array Indexing . https:\/\/docs.scipy.org\/doc\/numpy-1.13.0\/reference\/arrays.indexing.html , 2009 . NumPy Array Indexing. https:\/\/docs.scipy.org\/doc\/numpy-1.13.0\/reference\/arrays.indexing.html, 2009."},{"key":"e_1_2_1_45_1","unstructured":"NYC Taxi Dataset. https:\/\/cloud.google.com\/bigquery\/public-data\/nyc-tlc-trips.  NYC Taxi Dataset. https:\/\/cloud.google.com\/bigquery\/public-data\/nyc-tlc-trips."},{"key":"e_1_2_1_46_1","unstructured":"OpenMP. http:\/\/openmp.org\/wp\/.  OpenMP. http:\/\/openmp.org\/wp\/."},{"key":"e_1_2_1_47_1","first-page":"293","volume-title":"Chun. Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15)","author":"Ousterhout K.","year":"2015","unstructured":"K. Ousterhout , R. Rasti , S. Ratnasamy , S. Shenker , and B.- G. Chun. Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) , pages 293 -- 307 , 2015 . K. Ousterhout, R. Rasti, S. Ratnasamy, S. Shenker, and B.-G. Chun. Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 293--307, 2015."},{"key":"e_1_2_1_48_1","volume-title":"CIDR","author":"Palkar S.","year":"2017","unstructured":"S. Palkar , J. J. Thomas , A. Shanbhag , D. Narayanan , H. Pirk , M. Schwarzkopf , S. Amarasinghe , and M. Zaharia . Weld: A Common Runtime for High Performance Analytics . In CIDR , 2017 . S. Palkar, J. J. Thomas, A. Shanbhag, D. Narayanan, H. Pirk, M. Schwarzkopf, S. Amarasinghe, and M. Zaharia. Weld: A Common Runtime for High Performance Analytics. In CIDR, 2017."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3007328.3007336"},{"key":"e_1_2_1_50_1","volume-title":"https:\/\/pypi.python.org","year":"2018","unstructured":"Pypi. https:\/\/pypi.python.org , 2018 . Pypi. https:\/\/pypi.python.org, 2018."},{"key":"e_1_2_1_51_1","volume-title":"http:\/\/pytorch.org","year":"2018","unstructured":"Pytorch. http:\/\/pytorch.org , 2018 . Pytorch. http:\/\/pytorch.org, 2018."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.14778\/1453856.1453924"},{"key":"e_1_2_1_53_1","unstructured":"311\n    Service Requests Dataset. https:\/\/github.com\/jvns\/pandas-cookbook\/blob\/master\/data\/311-service-requests.csv.  311 Service Requests Dataset. https:\/\/github.com\/jvns\/pandas-cookbook\/blob\/master\/data\/311-service-requests.csv."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2429069.2429128"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/42201.42203"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915244"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2010.69"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.14778\/2809974.2809983"},{"key":"e_1_2_1_59_1","volume-title":"https:\/\/www.tensorflow.org\/performance\/xla\/","author":"TensorFlow","year":"2018","unstructured":"TensorFlow XLA. https:\/\/www.tensorflow.org\/performance\/xla\/ , 2018 . TensorFlow XLA. https:\/\/www.tensorflow.org\/performance\/xla\/, 2018."},{"key":"e_1_2_1_60_1","first-page":"1","volume-title":"Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08","author":"Yu Y.","year":"2008","unstructured":"Y. Yu , M. Isard , D. Fetterly , M. Budiu , U. Erlingsson , P. K. Gunda , and J. Currey . DryadLINQ: A System for General-purpose Distributed Data-parallel Computing Using a High-level Language . In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08 , pages 1 -- 14 , Berkeley, CA, USA , 2008 . USENIX Association. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-purpose Distributed Data-parallel Computing Using a High-level Language. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08, pages 1--14, Berkeley, CA, USA, 2008. USENIX Association."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2934664"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3213880.3213890","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T05:01:29Z","timestamp":1672203689000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3213880.3213890"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5]]},"references-count":61,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2018,5]]}},"alternative-id":["10.14778\/3213880.3213890"],"URL":"https:\/\/doi.org\/10.14778\/3213880.3213890","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2018,5]]}}}