{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T10:10:15Z","timestamp":1764843015660,"version":"3.44.0"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"name":"European Union\u2019s Horizon 2020\/EuroHPC research and innovation programme","award":["955606 (DEEP-SEA), and PN038700 (Zettascale Laboratory)"],"award-info":[{"award-number":["955606 (DEEP-SEA), and PN038700 (Zettascale Laboratory)"]}]},{"name":"Spanish State Research Agency - Ministry of Science and Innovation","award":["PID2019-107255GB-C21\/MCIN\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["PID2019-107255GB-C21\/MCIN\/AEI\/10.13039\/501100011033"]}]},{"name":"Ramon y Cajal fellowship","award":["RYC2018-025628-I\/MCIN\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["RYC2018-025628-I\/MCIN\/AEI\/10.13039\/501100011033"]}]},{"DOI":"10.13039\/501100002809","name":"Generalitat de Catalunya","doi-asserted-by":"crossref","award":["2017-SGR-1414"],"award-info":[{"award-number":["2017-SGR-1414"]}],"id":[{"id":"10.13039\/501100002809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>Distributed tasking models such as OmpSs-2@Cluster, StarPU-MPI, and PaRSEC express HPC applications as task graphs with explicit dependencies. The single task graph unifies the representation of parallelism across CPU cores, accelerators, and distributed-memory nodes, offering higher programmer productivity compared to traditional MPI + X. Most task-based models construct the task graph sequentially, which provides a clear and familiar programming model, simplifying code development, maintenance, and porting. However, this design introduces a bottleneck in task creation and dependency management, limiting performance and scalability. As a result, unless the tasks are very coarse-grained, current distributed sequential tasking models cannot match the performance of MPI + X.<\/jats:p>\n          <jats:p>\n            Many scientific applications, however, are iterative in nature, constructing the same directed acyclic task graph at each timestep. We exploit this structure to eliminate the sequential bottleneck and control message overhead in a sequentially-constructed distributed tasking model, while preserving its simplicity and productivity. Our approach builts on the recently proposed\n            <jats:sans-serif>taskiter<\/jats:sans-serif>\n            directive for OpenMP and OmpSs-2, allowing a single iteration to be expressed as a cyclic graph. The runtime partitions the cyclic graph across nodes, precomputes the MPI transfers, and then executes the loop body at low overhead. By integrating the MPI communications directly into the application\u2019s task graph, our approach naturally overlaps computation and communication, in some cases exposing dramatically more parallelism than fork\u2013join MPI + OpenMP. We define the programming model and describe the full runtime implementation, and integrate our proposal into OmpSs-2@Cluster. We evaluate it using five benchmarks on up to 128 nodes of the MareNostrum\u00a05 supercomputer. For applications with fork\u2013join parallelism, our approach has performance similar to fork\u2013join MPI + OpenMP, making it a viable productive alternative, unlike the existing OmpSs-2@Cluster model, which is up to 7.7 times slower than MPI + OpenMP. For a 2D Gauss\u2013Seidel stencil computation, our approach enables 3D wavefront computation, giving performance up to 22 times faster than fork\u2013join MPI + OpenMP and on-a-par with state-of-the-art TAMPI + OmpSs-2. All software, comprising the compiler, runtime, and benchmarks, is released open source.\n            <jats:xref ref-type=\"fn\">\n              <jats:sup>1<\/jats:sup>\n            <\/jats:xref>\n          <\/jats:p>","DOI":"10.1145\/3743134","type":"journal-article","created":{"date-parts":[[2025,6,5]],"date-time":"2025-06-05T07:12:13Z","timestamp":1749107533000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Leveraging iterative applications to improve the scalability of task-based programming models on distributed systems"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4410-5317","authenticated-orcid":false,"given":"Omar","family":"Shaaban Ibrahim ali","sequence":"first","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6575-6989","authenticated-orcid":false,"given":"Juliette","family":"Fournis d'Albiat","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6106-6978","authenticated-orcid":false,"given":"Isabel","family":"Piedrahita","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3580-9630","authenticated-orcid":false,"given":"Vicen\u00e7","family":"Beltran","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0417-3430","authenticated-orcid":false,"given":"Xavier","family":"Martorell","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9392-0521","authenticated-orcid":false,"given":"Paul","family":"Carpenter","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5146-103X","authenticated-orcid":false,"given":"Eduard","family":"Ayguad\u00e9","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7489-4727","authenticated-orcid":false,"given":"Jesus","family":"Labarta","sequence":"additional","affiliation":[{"name":"Computer Sciences, Barcelona Supercomputing Center","place":["Barcelona, Spain"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,17]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2022. The Chapel Parallel Programming Language. (2022). Retrieved April 4 2022 from https:\/\/chapel-lang.org\/"},{"key":"e_1_3_2_3_2","unstructured":"2024. Clang: A C Language Family Frontend for LLVM. (2024). Retrieved January 18 2024 from https:\/\/clang.llvm.org\/"},{"key":"e_1_3_2_4_2","unstructured":"2024. The LLVM Compiler Infrastructure. (2024). Retrieved January 18 2024 from https:\/\/llvm.org\/"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-12597-3_20"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545008.3545045"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Emmanuel Agullo Olivier Aumage Mathieu Faverge Nathalie Furmento Florent Pruvost Marc Sergent and Samuel Paul Thibault. 2017. Achieving high performance on supercomputers with a sequential task-based programming model. IEEE Transactions on Parallel and Distributed Systems (2017). 10.1109\/TPDS.2017.2766064","DOI":"10.1109\/TPDS.2017.2766064"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","unstructured":"David \u00c1lvarez and Vicen\u00e7 Beltran. 2023. Optimizing iterative data-flow scientific applications using directed cyclic graphs. IEEE Access 11 (2023) 51971\u201351984. 10.1109\/ACCESS.2023.3269902","DOI":"10.1109\/ACCESS.2023.3269902"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437801.3441601"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33518-1_40"},{"key":"e_1_3_2_11_2","unstructured":"Barcelona Supercomputing Center. 2021. Influence in OpenMP - OmpSs-2 Specification. (2021). Retrieved July 7 2022 from https:\/\/pm.bsc.es\/ftp\/ompss-2\/doc\/spec\/introduction"},{"key":"e_1_3_2_12_2","unstructured":"Barcelona Supercomputing Center. 2021. Nanos6. https:\/\/github.com\/bsc-pm\/nanos6. Accessed: 2025-04-01."},{"key":"e_1_3_2_13_2","unstructured":"Barcelona Supercomputing Center. 2021. OmpSs-2 Specification. https:\/\/pm.bsc.es\/ftp\/ompss-2\/doc\/spec\/. Accessed: 2025-04-01."},{"key":"e_1_3_2_14_2","unstructured":"Barcelona Supercomputing Center. 2022. OmpSs-2@Cluster releases. https:\/\/github.com\/bsc-pm\/ompss-2-cluster-releases. Accessed: 2023-07-06."},{"key":"e_1_3_2_15_2","unstructured":"Barcelona Supercomputing Center. 2024. MareNostrum 5 System Architecture. https:\/\/www.bsc.es\/marenostrum\/marenostrum-5. Accessed: 2025-04-01."},{"key":"e_1_3_2_16_2","unstructured":"Barcelona Supercomputing Center. 2024. OmpSs-2 LLVM Compiler Infrastructure. https:\/\/github.com\/bsc-pm\/llvm. Accessed: 2024-02-06."},{"key":"e_1_3_2_17_2","unstructured":"Barcelona Supercomputing Center. 2025. Taskiter. https:\/\/github.com\/bsc-pm\/ompss-2-cluster-releases#taskiter. Accessed: 2025-04-01."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2012.71"},{"key":"e_1_3_2_19_2","volume-title":"Dense Linear Algebra on Distributed Heterogeneous Hardware with a Symbolic DAG Approach","author":"Bosilca George","year":"2012","unstructured":"George Bosilca, Aurelien Bouteiller, Anthony Danalis, Thomas Herault, Pitior Luszczek, and Jack Dongarra. 2012. Dense Linear Algebra on Distributed Heterogeneous Hardware with a Symbolic DAG Approach. Technical Report. Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States). Retrieved fromhttps:\/\/www.osti.gov\/servlets\/purl\/1173290"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2012.58"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Paul Cardosi and B\u00e9renger Bramas. 2024. Specx: A C++ task-based runtime system for heterogeneous distributed architectures. arXiv:2308.15964 [cs.DC] https:\/\/arxiv.org\/abs\/2308.15964","DOI":"10.7717\/peerj-cs.2966"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2020373.2020375"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/1094811.1094852"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2013.78"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.1995.375471"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS53621.2022.00085"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","unstructured":"Ewa Deelman Gurmeet Singh Mei-Hui Su James Blythe Yolanda Gil Carl Kesselman Gaurang Mehta Karan Vahi G. Berriman John Good Anastasia Laity and Daniel S. Katz. 2005. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming 13 3 (2005) 219\u2013237. 10.1155\/2005\/128026","DOI":"10.1155\/2005\/128026"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","unstructured":"Alejandro Duran Eduard Ayguad\u00e9 Rosa M. Badia Jes\u00fas Labarta Luis Martinell Xavier Martorell and Judit Planas. 2011. OmpSs: A proposal for programming heterogeneous multi-core architectures. Parallel Processing Letters 21 2 (2011) 173\u2013193. 10.1142\/S0129626411000151","DOI":"10.1142\/S0129626411000151"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Karl F\u00fcrlinger Jos\u00e9 Gracia Andreas Kn\u00fcpfer Tobias Fuchs Denis H\u00fcnich Pascal Jungblut Roger Kowalewski and Joseph Schuchart. 2020. DASH: Distributed data structures and parallel algorithms in a global address space. In Software for Exascale Computing-SPPEXA 2016-2019. Springer International Publishing. 10.1007\/978-3-030-47956-5_6","DOI":"10.1007\/978-3-030-47956-5_6"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the 7th International Conference on PGAS Programming Models","author":"Gr\u00fcnewald Daniel","year":"2013","unstructured":"Daniel Gr\u00fcnewald and Christian Simmendinger. 2013. The GASPI API specification and its implementation GPI 2.0. In Proceedings of the 7th International Conference on PGAS Programming Models."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3148226.3148233"},{"key":"e_1_3_2_32_2","unstructured":"Gabriele Jost Hao-Qiang Jin and Ferhat F. Hatay. 2003. Comparing the OpenMP MPI and hybrid programming paradigm on an SMP cluster. In European Workshop on OpenMP and Applications (2003). https:\/\/ntrs.nasa.gov\/citations\/20030107321"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.13140\/2.1.2635.5204"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","unstructured":"Jannis Klinkenberg Philipp Samfass Michael Bader Christian Terboven and Matthias M\u00fcller. 2019. CHAMELEON: Reactive load balancing for hybrid MPI+OpenMP task-parallel applications. J. Parallel and Distrib. Comput. 138 C (2019) 55\u201364. 10.1016\/j.jpdc.2019.12.005","DOI":"10.1016\/j.jpdc.2019.12.005"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29737-3_48"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","unstructured":"Francesc Lordan Enric Tejedor Jorge Ejarque Roger Rafanell Javier Alvarez Fabrizio Marozzo Daniele Lezzi Ra\u00fcl Sirvent Domenico Talia and Rosa M. Badia. 2014. ServiceSs: An interoperable programming framework for the cloud. Journal of Grid Computing 12 1 (2014) 67\u201391. 10.1007\/s10723-013-9272-5","DOI":"10.1007\/s10723-013-9272-5"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISBI.2008.4541126"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.5821\/dissertation-2117-380814"},{"key":"e_1_3_2_39_2","unstructured":"MPI Forum. 2023. MPI Documents. (2023). Retrieved February 19 2024 from https:\/\/www.mpi-forum.org\/docs\/"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCW63240.2024.00239"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","unstructured":"Robert W. Numrich and John Reid. 1998. Co-Array Fortran for parallel programming. In ACM SIGPLAN Fortran Forum. ACM New York NY USA 17 (1998) 1\u201331. 10.1145\/289918.289920","DOI":"10.1145\/289918.289920"},{"key":"e_1_3_2_42_2","unstructured":"OpenACC Organization. 2011. OpenACC: Directives for Accelerators. (2011). Retrieved February 2 2024 from http:\/\/www.openacc-standard.org"},{"key":"e_1_3_2_43_2","unstructured":"OpenMP Architecture Review Board. 2021. OpenMP Application Programming Interface Version 5.2. (2021). Retrieved April 19 2022 from https:\/\/www.openmp.org\/wp-content\/uploads\/OpenMP-API-Specification-5-2.pdf"},{"key":"e_1_3_2_44_2","unstructured":"Parallel Programming Lab Dept of Computer Science University of Illinois. 2023. Charm++ Documentation. (2023). Retrieved July 6 2023 from https:\/\/charm.readthedocs.io\/en\/latest"},{"key":"e_1_3_2_45_2","volume-title":"High Performance Computing","author":"Patel Atmn","year":"2022","unstructured":"Atmn Patel and Johannes Doerfert. 2022. Remote OpenMP offloading. In High Performance Computing. Ana-Lucia Varbanescu, Abhinav Bhatele, Piotr Luszczek, and Baboulin Marc (Eds.), Springer International Publishing, Cham."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2017.69"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTR.2008.4663765"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-27170-8_31"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","unstructured":"Pedro Henrique Rosso Lucian Petrica Nusrat Jahan Lisa Marcio Pereira Sandro Rigo Herv\u00e9 Yviquel Vanderlei Bonato Emilio Francesquini and Guido Araujo. 2024. Integrating multi-FPGA acceleration to OpenMP distributed computing. In Advancing OpenMP for Future Accelerators Alexis Espinosa Michael Klemm Bronis R. de Supinski Maciej Cytowski and Jannis Klinkenberg (Eds.). Springer Nature Switzerland 49\u201363.","DOI":"10.1007\/978-3-031-72567-8_4"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-54420-0_5"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/Cluster48925.2021.00024"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","unstructured":"Kevin Sala Xavier Teruel Josep M. Perez Antonio J. Pe\u00f1a Vicen\u00e7 Beltran and Jesus Labarta. 2019. Integrating blocking and non-blocking MPI primitives with task-based programming models. Parallel Comput. 85 (2019) 153\u2013166. 10.1016\/j.parco.2018.12.008","DOI":"10.1016\/j.parco.2018.12.008"},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-981-15-7683-6","volume-title":"XcalableMP PGAS Programming Language: From Programming Model to Applications","author":"Sato Mitsuhisa","year":"2021","unstructured":"Mitsuhisa Sato. 2021. XcalableMP PGAS Programming Language: From Programming Model to Applications. Springer Nature."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582514.3582519"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807629"},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"Lorna Smith and Mark Bull. 2001. Development of mixed mode MPI\/OpenMP applications. Sci. Program. 9 2\u20133 (2001) 83\u201398.","DOI":"10.1155\/2001\/450503"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/2312005.2312025"},{"key":"e_1_3_2_58_2","unstructured":"UPC Consortium. 2024. Berkeley UPC \u2013 Unified Parallel C. (2024). Retrieved February 12 2024 from https:\/\/upc.lbl.gov\/"},{"key":"e_1_3_2_59_2","article-title":"Chapel, Fortress and X10: Novel Languages for HPC","author":"Weiland Michele","year":"2007","unstructured":"Michele Weiland. 2007. Chapel, Fortress and X10: Novel Languages for HPC. EPCC, The University of Edinburgh, Technical Report HPCxTR0706.","journal-title":"EPCC, The University of Edinburgh, Technical Report HPCxTR0706"},{"key":"e_1_3_2_60_2","volume-title":"Dynamic Task Execution on Shared and Distributed Memory Architectures","author":"YarKhan Asim","year":"2012","unstructured":"Asim YarKhan. 2012. Dynamic Task Execution on Shared and Distributed Memory Architectures. Ph.D. Dissertation. University of Tennessee."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","unstructured":"Chenle Yu Sara Royuela and Eduardo Quinones. 2023. Taskgraph: A low contention openmp tasking framework. IEEE Transactions on Parallel and Distributed Systems 34 8 (2023) 2325\u20132336. 10.1109\/TPDS.2023.3284219","DOI":"10.1109\/TPDS.2023.3284219"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3547276.3548444"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3743134","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T13:43:31Z","timestamp":1758116611000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3743134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,17]]},"references-count":61,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3743134"],"URL":"https:\/\/doi.org\/10.1145\/3743134","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"type":"print","value":"1544-3566"},{"type":"electronic","value":"1544-3973"}],"subject":[],"published":{"date-parts":[[2025,9,17]]},"assertion":[{"value":"2024-05-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}