{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T01:13:54Z","timestamp":1789434834747,"version":"build-2803163510"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","license":[{"start":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T00:00:00Z","timestamp":1728345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2024,10,8]]},"abstract":"<jats:p>Data science workloads frequently include Python code, but Python\u2019s dynamic nature makes efficient execution hard. Traditional approaches either treat Python as a black box, missing out on optimization potential, or are limited to a narrow domain. However, a deep and efficient integration of user-defined Python code into data processing systems requires extracting the semantics of the entire Python code.<\/jats:p>\n                  <jats:p>In this paper, we propose a novel approach for extracting the high-level semantics by transforming general Python functions into program generators that generate a statically-typed IR when executed. The extracted IR then allows for high-level, domain-specific optimizations and the generation of efficient C++ code. With our prototype implementation, HiPy, we achieve single-threaded speedups of 2\u201320x for many workloads. Furthermore, HiPy is also capable of accelerating Python code in other domains like numerical data, where it can sometimes even outperform specialized compilers.<\/jats:p>","DOI":"10.1145\/3689737","type":"journal-article","created":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T03:23:04Z","timestamp":1728357784000},"page":"736-762","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["HiPy: Extracting High-Level Semantics from Python Code for Data Processing"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0890-1914","authenticated-orcid":false,"given":"Michael","family":"Jungmair","sequence":"first","affiliation":[{"name":"TU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1900-1292","authenticated-orcid":false,"given":"Alexis","family":"Engelke","sequence":"additional","affiliation":[{"name":"TU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1926-3551","authenticated-orcid":false,"given":"Jana","family":"Giceva","sequence":"additional","affiliation":[{"name":"TU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,10,8]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"[n. d.]. TorchScript. https:\/\/pytorch.org\/docs\/stable\/jit.html. Accessed: 2024-03-27."},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356549"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.ASCOM.2014.12.001"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2010.118"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1137\/141000671"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611554"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","unstructured":"Hassan Chafi Zach DeVito Adriaan Moors Tiark Rompf Arvind K. Sujeeth Pat Hanrahan Martin Odersky and Kunle Olukotun. 2010. Language virtualization for heterogeneous parallel computing. In Proceedings of the 25th Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems Languages and Applications (OOPSLA 2010). 835\u2013847. https:\/\/doi.org\/10.1145\/1869459.1869527 10.1145\/1869459.1869527","DOI":"10.1145\/1869459.1869527"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459010"},{"key":"e_1_3_1_10_2","unstructured":"James M. Decker Dan Moldovan Andrew A. Johnson Guannan Wei Vritant Bhardwaj Gregory Essertel Fei Wang Alexander B. Wiltschko and Tiark Rompf. 2019. Snek: Overloading Python Semantics via Virtualization."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.14778\/3547305.3547328"},{"issue":"5","key":"e_1_3_1_12_2","first-page":"45","article-title":"Partial evaluation of computation process-an approach to a compilercompiler","volume":"2","author":"Futamura Yoshihiko","year":"1971","unstructured":"Yoshihiko Futamura. 1971. Partial evaluation of computation process-an approach to a compilercompiler. Systems, computers, controls 2, 5 (1971), 45\u201350.","journal-title":"Systems, computers, controls"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.14778\/3489496.3489501"},{"key":"e_1_3_1_14_2","unstructured":"Stefan Hagedorn Steffen Kl\u00e4be and Kai-Uwe Sattler. 2021. Putting Pandas in a Box. In 11th Conference on Innovative Data Systems Research CIDR 2021 Virtual Event January 11-15 2021 Online Proceedings. www.cidrdb.org. http:\/\/cidrdb.org\/cidr2021\/papers\/cidr2021_paper07.pdf"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/153676"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/512644.512652"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","unstructured":"Michael Jungmair Alexis Engelke and Jana Giceva. 2024. Artifact for \u201cHiPy: Extracting High-Level Semantics From Python Code For Data Processing\u201d. https:\/\/doi.org\/10.5281\/zenodo.13323059 10.5281\/zenodo.13323059","DOI":"10.5281\/zenodo.13323059"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","unstructured":"Shoaib Kamil Alvin Cheung Shachar Itzhaky and Armando Solar-Lezama. 2016. Verified lifting of stencil computations. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI \u201916). 711\u2013726. https:\/\/doi.org\/10.1145\/2908080.2908117 10.1145\/2908080.2908117","DOI":"10.1145\/2908080.2908117"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2833157.2833162"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","unstructured":"Chris Lattner Mehdi Amini Uday Bondhugula Albert Cohen Andy Davis Jacques A. Pienaar River Riddle Tatiana Shpeisman Nicolas Vasilache and Oleksandr Zinenko. 2021. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation. In IEEE\/ACM International Symposium on Code Generation and Optimization (CGO 2021). 2\u201314. https:\/\/doi.org\/10.1109\/CGO51591.2021.9370308 10.1109\/CGO51591.2021.9370308","DOI":"10.1109\/CGO51591.2021.9370308"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.25080\/MAJORA-92BF1922-00A"},{"key":"e_1_3_1_22_2","unstructured":"Modular Inc. 2024. Mojo. https:\/\/www.modular.com\/mojo. Accessed: 2024-08-29."},{"key":"e_1_3_1_23_2","volume-title":"Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 - April 2, 2019","author":"Moldovan Dan","year":"2019","unstructured":"Dan Moldovan, James M. Decker, Fei Wang, Andrew A. Johnson, Brian K. Lee, Zachary Nado, D. Sculley, Tiark Rompf, and Alexander B. Wiltschko. 2019. AutoGraph: Imperative-style Coding with Graph-based Performance. In Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 - April 2, 2019. mlsys.org. https:\/\/proceedings.mlsys.org\/book\/272.pdf"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.14778\/2002938.2002940"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00023"},{"key":"e_1_3_1_26_2","unstructured":"Nuitka Team. 2023. Nuitka: A Python compiler written in Python. https:\/\/github.com\/Nuitka\/Nuitka."},{"key":"e_1_3_1_27_2","unstructured":"Oracle Labs. 2023. GraalPython: A Python 3 Implementation Built on GraalVM. https:\/\/www.graalvm.org\/python\/. Accessed: 2024-03-27."},{"key":"e_1_3_1_28_2","volume-title":"8th Biennial Conference on Innovative Data Systems Research, CIDR 2017, Chaminade, CA, USA, January 8-11, 2017, Online Proceedings","author":"Palkar Shoumik","year":"2017","unstructured":"Shoumik Palkar, James Thomas, Anil Shanbhag, Malte Schwarzkopf, Saman P. Amarasinghe, and Matei Zaharia. 2017. Weld: A Common Runtime for High Performance Data Analysis. In 8th Biennial Conference on Innovative Data Systems Research, CIDR 2017, Chaminade, CA, USA, January 8-11, 2017, Online Proceedings. www.cidrdb.org. http:\/\/cidrdb.org\/cidr2017\/papers\/p127-palkar-cidr17.pdf"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.14778\/3494124.3494152"},{"key":"e_1_3_1_30_2","unstructured":"Pyston Team. 2023. Pyston: A faster and highly-compatible implementation of the Python programming language. https:\/\/github.com\/pyston\/pyston."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","unstructured":"Mark Raasveldt and Hannes M\u00fchleisen. 2016. Vectorized UDFs in Column-Stores. In Proceedings of the 28th International Conference on Scientific and Statistical Database Management (SSDBM \u201916). Article 16. https:\/\/doi.org\/10.1145\/2949689.2949703 10.1145\/2949689.2949703","DOI":"10.1145\/2949689.2949703"},{"key":"e_1_3_1_32_2","first-page":"638","volume-title":"Proceedings of Machine Learning and Systems","author":"Reed James K.","year":"2022","unstructured":"James K. Reed, Zachary DeVito, Horace He, Ansley Ussery, and Jason Ansel. 2022. torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. In Proceedings of Machine Learning and Systems. mlsys.org, 638\u2013651. https:\/\/proceedings.mlsys.org\/paper\/2022\/hash\/ca46c1b9512a7a8315fa3c5a946e8265-Abstract.html"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","unstructured":"Armin Rigo and Samuele Pedroni. 2006. PyPy\u2019s approach to virtual machine construction. In Companion to the 21th Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems Languages and Applications (OOPSLA \u201806). 944\u2013953. https:\/\/doi.org\/10.1145\/1176617.1176753 10.1145\/1176617.1176753","DOI":"10.1145\/1176617.1176753"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2184319.2184345"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555041.3589735"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3578360.3580275"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","unstructured":"Phanwadee Sinthong and Michael J. Carey. 2019. AFrame: Extending DataFrames for Large-Scale Modern Data Analysis. In 2019 IEEE International Conference on Big Data (IEEE BigData). 359\u2013371. https:\/\/doi.org\/10.1109\/BIGDATA47090.2019.9006303 10.1109\/BIGDATA47090.2019.9006303","DOI":"10.1109\/BIGDATA47090.2019.9006303"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476281"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457244"},{"issue":"10","key":"e_1_3_1_40_2","first-page":"95","article-title":"Spark: Cluster computing with working sets","volume":"10","author":"Zaharia Matei","year":"2010","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, Ion Stoica, et al. 2010. Spark: Cluster computing with working sets. HotCloud 10, 10-10 (2010), 95.","journal-title":"HotCloud"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3689737","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3689737","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T09:08:19Z","timestamp":1770196099000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3689737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,8]]},"references-count":39,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2024,10,8]]}},"alternative-id":["10.1145\/3689737"],"URL":"https:\/\/doi.org\/10.1145\/3689737","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,8]]},"assertion":[{"value":"2024-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}