{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T05:23:48Z","timestamp":1740893028178,"version":"3.38.0"},"reference-count":50,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2020,7,14]],"date-time":"2020-07-14T00:00:00Z","timestamp":1594684800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/501100002943","name":"Departament d\u2019Innovaci\u00f3, Universitats i Empresa, Generalitat de Catalunya","doi-asserted-by":"publisher","award":["2014-SGR-1051"],"award-info":[{"award-number":["2014-SGR-1051"]}],"id":[{"id":"10.13039\/501100002943","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:p> The last improvements in programming languages and models have focused on simplicity and abstraction; leading Python to the top of the list of the programming languages. However, there is still room for improvement when preventing users from dealing directly with distributed and parallel computing issues. This paper proposes and evaluates AutoParallel, a Python module to automatically find an appropriate task-based parallelisation of affine loop nests and execute them in parallel in a distributed computing infrastructure. It is based on sequential programming and contains one single annotation (in the form of a Python decorator) so that anyone with intermediate-level programming skills can scale up an application to hundreds of cores. <\/jats:p><jats:p> The evaluation demonstrates that AutoParallel goes one step further in easing the development of distributed applications. On the one hand, the programmability evaluation highlights the benefits of using a single Python decorator instead of manually annotating each task and its parameters or, even worse, having to develop the parallel code explicitly (e.g., using OpenMP, MPI). On the other hand, the performance evaluation demonstrates that AutoParallel is capable of automatically generating task-based workflows from sequential Python code while achieving the same performances than manually taskified versions of established state-of-the-art algorithms (i.e., Cholesky, LU, and QR decompositions). Finally, AutoParallel is also capable of automatically building data blocks to increase the tasks\u2019 granularity; freeing the user from creating the data chunks, and re-designing the algorithm. For advanced users, we believe that this feature can be useful as a baseline to design blocked algorithms. <\/jats:p>","DOI":"10.1177\/1094342020937050","type":"journal-article","created":{"date-parts":[[2020,7,14]],"date-time":"2020-07-14T15:07:05Z","timestamp":1594739225000},"page":"659-675","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":3,"title":["AutoParallel: Automatic parallelisation and distributed execution of affine loop nests in Python"],"prefix":"10.1177","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4170-818X","authenticated-orcid":false,"given":"Cristian","family":"Ramon-Cortes","sequence":"first","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}]},{"given":"Ramon","family":"Amela","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4725-5097","authenticated-orcid":false,"given":"Jorge","family":"Ejarque","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}]},{"given":"Philippe","family":"Clauss","sequence":"additional","affiliation":[{"name":"INRIA - ICube Lab \u2013 Universit\u00e9 de Strasbourg, Strasbourg, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2941-5499","authenticated-orcid":false,"given":"Rosa M.","family":"Badia","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}]}],"member":"179","published-online":{"date-parts":[[2020,7,14]]},"reference":[{"key":"bibr1-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/3149869.3149870"},{"key":"bibr2-1094342020937050","doi-asserted-by":"publisher","DOI":"10.2516\/ogst\/2018047"},{"key":"bibr3-1094342020937050","unstructured":"Anaconda (2020) Numba: a high performance python compiler. Available at: http:\/\/numba.pydata.org (accessed 8 April 2020)."},{"key":"bibr4-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719604"},{"key":"bibr5-1094342020937050","unstructured":"Apache Software Fundation (2019) PySpark. Available at: https:\/\/spark.apache.org\/docs\/latest\/api\/python\/index.html (accessed 8 October 2019)."},{"key":"bibr6-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1016\/j.softx.2015.10.004"},{"key":"bibr7-1094342020937050","unstructured":"Barcelona Supercomputing Center (BSC) (2019a) COMPSs GitHub. Available at: https:\/\/github.com\/bsc-wdc\/compss. (accessed 8 October 2019)."},{"key":"bibr8-1094342020937050","unstructured":"Barcelona Supercomputing Center (BSC) (2019b) Extrae Tool. Available at: https:\/\/tools.bsc.es\/extrae (accessed 9 October 2019)."},{"key":"bibr9-1094342020937050","unstructured":"Barcelona Supercomputing Center (BSC) (2019c) MareNostrum IV Technical Information. Available at: https:\/\/www.bsc.es\/marenostrum\/marenostrum\/technical-information (accessed 8 October 2019)."},{"key":"bibr10-1094342020937050","unstructured":"Barcelona Supercomputing Center (BSC) (2019d) Paraver Tool. Available at: https:\/\/tools.bsc.es\/paraver (accessed 8 October 2019)."},{"key":"bibr11-1094342020937050","unstructured":"Barcelona Supercomputing Center (BSC) (2020) PyCOMPSs User Manual. Available at: https:\/\/compss-doc.readthedocs.io\/en\/2.6\/Sections\/02_User_Manual_App_Development.html#python-binding (accessed 8 April 2020)."},{"key":"bibr12-1094342020937050","doi-asserted-by":"crossref","unstructured":"Bastoul C (2004) Code generation in the polyhedral model is easier than you think. In: Proceedings. 13th International Conference on Parallel Architecture and Compilation Techniques. PACT 2004. USA: IEEE Computer Society, pp. 7\u201316.","DOI":"10.1109\/PACT.2004.1342537"},{"key":"bibr13-1094342020937050","unstructured":"Bastoul C (2011) OpenScop: A Specification and a Library for Data Exchange in Polyhedral Compilation Tools. Technical Report, Paris-Sud University, France. Available at: http:\/\/icps.u-strasbg.fr\/people\/bastoul\/public_html\/development\/openscop\/docs\/openscop.html (accessed 8 October 2019; updated September 2011)."},{"key":"bibr14-1094342020937050","doi-asserted-by":"crossref","unstructured":"Bastoul C, Cohen A, Girbal1 S, et al. (2003) Putting Polyhedral Loop Transformations to Work. In: International Workshop on Languages and Compilers for Parallel Computing. Springer Berlin Heidelberg: Springer, pp. 209\u2013225.","DOI":"10.1007\/978-3-540-24644-2_14"},{"issue":"1","key":"bibr15-1094342020937050","volume":"35","author":"Bientinesi P","year":"2008","journal-title":"ACM Transactions on Mathematical Softwar"},{"key":"bibr51-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/567806.567807"},{"key":"bibr16-1094342020937050","unstructured":"Bondhugula U (2017) Pluto. Available at: http:\/\/pluto-compiler.sourceforge.net (accessed 8 October 2019)."},{"key":"bibr17-1094342020937050","doi-asserted-by":"crossref","unstructured":"Bondhugula U, Hartono A, Ramanujam J, et al. (2008a) A practical automatic polyhedral Parallelizer and locality optimizer. SIGPLAN Not 43(6): 101\u2013113.","DOI":"10.1145\/1379022.1375595"},{"key":"bibr18-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78791-4_9"},{"key":"bibr19-1094342020937050","unstructured":"Cass S (2019) The top programming languages 2019: python remains the big kahuna, but specialist languages hold their own. Available at: https:\/\/spectrum.ieee.org\/computing\/software\/the-top-programming-languages-2019 (accessed 19 December 2019)."},{"key":"bibr20-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/1088149.1088169"},{"key":"bibr21-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1177\/1094342017701278"},{"key":"bibr22-1094342020937050","unstructured":"Cooke DM, Alted F, et al. (2020) NumExpr: fast numerical expression evaluator for NumPy. Available at: https:\/\/github.com\/pydata\/numexpr (accessed 8 April 2020)."},{"key":"bibr23-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1109\/99.660313"},{"key":"bibr24-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2005.03.010"},{"key":"bibr25-1094342020937050","unstructured":"Dask Development Team (2016) Dask: library for dynamic task scheduling. Available at: http:\/\/dask.pydata.org (accessed 19 December 2019; updated 2019)."},{"key":"bibr26-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/131766.131769"},{"key":"bibr27-1094342020937050","volume-title":"Matrix Computations","author":"Golub GH","year":"1996","edition":"3"},{"key":"bibr28-1094342020937050","unstructured":"Google (2019) The Go programming language. Available at: https:\/\/golang.org\/ (accessed 8 October 2019)."},{"key":"bibr29-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/504210.504213"},{"key":"bibr30-1094342020937050","unstructured":"Intel (2019) Threading Building Blocks (Intel TBB). Available at: https:\/\/software.intel.com\/en-us\/tbb (accessed 8 October 2019)."},{"key":"bibr31-1094342020937050","unstructured":"Jones E, Oliphant T, Peterson P (2001\u2013) SciPy: Open source scientific tools for Python. Available at: http:\/\/www.scipy.org\/ (accessed 8 October 2019; updated May 2020)."},{"key":"bibr32-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/2833157.2833162"},{"key":"bibr33-1094342020937050","volume-title":"Java Native Interface: Programmer\u2019s Guide and Reference","author":"Liang S","year":"1999","edition":"1"},{"key":"bibr34-1094342020937050","doi-asserted-by":"crossref","unstructured":"Lordan F, Tejedor E, Ejarque J, et al. (2014) ServiceSs: an interoperable programming framework for the Cloud. Journal of Grid Computing 12(1): 67\u201391.","DOI":"10.1007\/s10723-013-9272-5"},{"key":"bibr35-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.4192"},{"key":"bibr36-1094342020937050","first-page":"1","volume":"14","author":"McKinney W","year":"2011","journal-title":"Python for High Performance and Scientific Computing"},{"key":"bibr37-1094342020937050","first-page":"645","volume-title":"11th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 14)","author":"M\u00fcller SC","year":"2014"},{"key":"bibr38-1094342020937050","unstructured":"Python Software Fundation (2019) Parallel Processing and Multiprocessing in Python. Available at: https:\/\/wiki.python.org\/moin\/ParallelProcessing (accessed 8 October 2019)."},{"key":"bibr39-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1109\/PDP.2008.37"},{"key":"bibr40-1094342020937050","unstructured":"Ramon-Cortes C (2019a) Experimentation GitHub. Available at: https:\/\/github.com\/cristianrcv\/pycompss-autoparallel\/tree\/master\/examples (asccessed 8 October 2019)."},{"key":"bibr41-1094342020937050","unstructured":"Ramon-Cortes C (2019b) PyCOMPSs AutoParallel Module GitHub. Available at: https:\/\/github.com\/cristianrcv\/pycompss-autoparallel (accessed 8 October 2019)."},{"key":"bibr42-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1007\/s10723-017-9425-z"},{"key":"bibr43-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1145\/2838734"},{"key":"bibr44-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1177\/1094342015594678"},{"key":"bibr45-1094342020937050","unstructured":"The Ohio State University (2015) PolyBench\/C: The Polyhedral Benchmark suite. Available at: http:\/\/web.cse.ohio-state.edu\/pouchet.2\/software\/polybench (accessed 8 October 2019)."},{"volume-title":"The Python Language Reference Manual","year":"2011","author":"van Rossum G","key":"bibr47-1094342020937050"},{"key":"bibr48-1094342020937050","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2011.37"},{"key":"bibr49-1094342020937050","unstructured":"Vanovschi V (2019) Parallel Python Software. Available at: http:\/\/www.parallelpython.com (accessed 8 October 2019)."},{"issue":"10","key":"bibr50-1094342020937050","first-page":"95","volume":"10","author":"Zaharia M","year":"2010","journal-title":"HotCloud"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342020937050","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342020937050","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342020937050","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T05:50:47Z","timestamp":1740808247000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342020937050"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,14]]},"references-count":50,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["10.1177\/1094342020937050"],"URL":"https:\/\/doi.org\/10.1177\/1094342020937050","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"type":"print","value":"1094-3420"},{"type":"electronic","value":"1741-2846"}],"subject":[],"published":{"date-parts":[[2020,7,14]]}}}