{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T15:02:44Z","timestamp":1779894164831,"version":"3.53.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"2","funder":[{"name":"(EuroHPC) TEXTAROSSA project","award":["956831"],"award-info":[{"award-number":["956831"]}]},{"name":"Spanish Government","award":["PCI2021-121964\u2013TEXTAROSSA and CEX2021-001148-S"],"award-info":[{"award-number":["PCI2021-121964\u2013TEXTAROSSA and CEX2021-001148-S"]}]},{"name":"MICIU\/AEI","award":["PID2023-147979NB-C21"],"award-info":[{"award-number":["PID2023-147979NB-C21"]}]},{"name":"FEDER, European Union","award":["PID2023-147979NB-C21"],"award-info":[{"award-number":["PID2023-147979NB-C21"]}]},{"DOI":"10.13039\/501100002809","name":"Generalitat de Catalunya","doi-asserted-by":"crossref","award":["2021 SGR 01007"],"award-info":[{"award-number":["2021 SGR 01007"]}],"id":[{"id":"10.13039\/501100002809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Barcelona Zettascale Laboratory"},{"name":"Ministry for Digital Transformation and the Civil Service"},{"name":"European Union \u2013 NextGenerationEU"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Reconfigurable Technol. Syst."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Field Programmable Gate Arrays (FPGA) have seen an increase in popularity in High-Performance Computing (HPC) environments due to their high flexibility and energy efficiency. However, the adoption of these devices for HPC application acceleration is relatively new, and their application is still focused on niche applications mainly due to the programmability issues FPGAs have traditionally faced. In this work, we present an open source toolchain with support for different FPGA devices and memory topologies, and for implicit and explicit message-passing multi-node and multi-FPGA programming models, which, to the best of our knowledge, is the first of its kind available. The presented toolchain tries to address the increased complexity of programming newer multi-die devices using High-Bandwidth Memory (HBM) by proposing and evaluating HBM design strategies as well as inter-FPGA communication. In addition to presenting the framework characteristics, we evaluate its performance results over a cluster of FPGAs for a set of HPC application benchmarks delivering performance improvements over previous implementations in both single and multi-node versions. The presented framework is an ongoing effort to solve the lack of mature tools and ecosystems that allow programming large clusters of FPGAs and consequently make scaling applications across multiple FPGA devices a simpler task.<\/jats:p>","DOI":"10.1145\/3786349","type":"journal-article","created":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T13:03:11Z","timestamp":1766494991000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["OmpSs@FPGA: An Open Source Framework for Programming FPGA Clusters"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6071-3254","authenticated-orcid":false,"given":"Antonio","family":"Filgueras","sequence":"first","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya, Barcelona, Spain and Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5375-324X","authenticated-orcid":false,"given":"Ismael el","family":"Basli","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7427-9118","authenticated-orcid":false,"given":"Juan Miguel de Haro","family":"Ruiz","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1973-8289","authenticated-orcid":false,"given":"Miquel","family":"Vidal","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0536-5183","authenticated-orcid":false,"given":"Carlos","family":"\u00c1lvarez","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya, Barcelona, Spain and Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6064-7883","authenticated-orcid":false,"given":"Daniel","family":"Jim\u00e9nez-Gonz\u00e1lez","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya, Barcelona, Spain and Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0417-3430","authenticated-orcid":false,"given":"Xavier","family":"Martorell","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya, Barcelona, Spain and Barcelona Supercomputing Center, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,5,27]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"cloudFPGA\/cFDK. 2024. The cloudFPGA Development Kit. Retrieved from https:\/\/github.com\/cloudFPGA\/cFDK"},{"key":"e_1_3_1_3_2","unstructured":"Alibaba Cloud. 2025. Alibaba FPGA as a Service. Retrieved from https:\/\/www.alibabacloud.com\/blog\/deep-dive-into-alibaba-cloud-f3-fpga-as-a-service-instances_594057"},{"key":"e_1_3_1_4_2","unstructured":"AWS 2025. Amazon F2 instances. Retrieved from https:\/\/aws.amazon.com\/en\/ec2\/instance-types\/f1\/"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/HOTI.2017.13"},{"key":"e_1_3_1_6_2","unstructured":"Advanced Micro Devices. 2023. Alveo U55C Data Center Accelerator Cards Data Sheet (DS978). Advanced Micro Devices. Retrieved from https:\/\/docs.amd.com\/r\/en-US\/ds978-u55c\/"},{"key":"e_1_3_1_7_2","unstructured":"Advanced Micro Devices. 2024. AXI High Bandwidth Memory Controller LogiCORE IP Product Guide (PG276). Advanced Micro Devices. Retrieved from https:\/\/docs.amd.com\/r\/en-US\/pg276-axi-hbm\/"},{"key":"e_1_3_1_8_2","unstructured":"AMD. 2024. UltraScale+ Devices Integrated 100G Ethernet Subsystem LogiCORE IP Product Guide (PG203). Retrieved from https:\/\/docs.amd.com\/r\/en-US\/pg203-cmac-usplus\/"},{"key":"e_1_3_1_9_2","unstructured":"ARM. 2023. AMBA AXI and ACE Protocol Specification Version E. Retrieved from https:\/\/developer.arm.com\/documentation\/ihi0022\/e"},{"key":"e_1_3_1_10_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. AIT User Guide. Retrieved from https:\/\/pm.bsc.es\/ftp\/ompss-2-at-fpga\/doc\/user-guide-git\/ait-options\/ait-help.html"},{"key":"e_1_3_1_11_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. OMPIF User Guide. Retrieved from https:\/\/pm.bsc.es\/ftp\/ompss-2-at-fpga\/doc\/user-guide-git\/nanos6-api.html#ompif-cluster-api"},{"key":"e_1_3_1_12_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. OmpSs@FPGA Accelerator Placement User Guide. Retrieved from https:\/\/pm.bsc.es\/ftp\/ompss-2-at-fpga\/doc\/user-guide-git\/ait-options\/placement-options.html"},{"key":"e_1_3_1_13_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. OmpSs@FPGA Clang Sources. Retrieved from https:\/\/github.com\/bsc-pm-ompss-at-fpga\/llvm"},{"key":"e_1_3_1_14_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. OmpSs@FPGA Cluster User Guide. Retrieved from https:\/\/pm.bsc.es\/ftp\/ompss-2-at-fpga\/doc\/user-guide-git\/run-OMPIF-programs.html"},{"key":"e_1_3_1_15_2","unstructured":"Programming Models Group at Barcelona Supercomputing Center. 2025. OmpSs@FPGA Framework Code. Retrieved from https:\/\/github.com\/bsc-pm-ompss-at-fpga"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3506713"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3332466.3374545"},{"key":"e_1_3_1_18_2","unstructured":"Barcelona Supercomputing Center. Supercomputing. 2024. MareNostrum Supercomputer. Barcelona Center. Retrieved March 31 2024 from https:\/\/www.bsc.es\/marenostrum\/marenostrum"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3431920.3439301"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3616497"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373087.3375296"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2021.3086106"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS53621.2022.00085"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3665283.3665292"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2024.06.041"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASAP.2018.8445106"},{"key":"e_1_3_1_27_2","unstructured":"Advanced Micro Devices. 2024. Alveo Card Management Solution Subsystem Product Guide. Retrieved from https:\/\/docs.amd.com\/r\/en-US\/pg348-cms-subsystem"},{"key":"e_1_3_1_28_2","unstructured":"Advanced Micro Devices. 2024. Vivado Design Suite User Guide: Implementation. Retrieved from https:\/\/docs.amd.com\/r\/en-US\/ug904-vivado-implementation\/"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCD.2017.74"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0129626411000151"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICFPT59805.2023.00048"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM57271.2023.00041"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2011.11"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC.2016.7761639"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.3390\/ijms231911408"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3609335"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3431920.3439289"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2554688.2554776"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11265-021-01640-8"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW52791.2021.00030"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM57271.2023.00023"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3431920.3439284"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3517131"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2902961.2903025"},{"key":"e_1_3_1_45_2","unstructured":"Message Passing Interface Forum. 2021. MPI: A Message-Passing Interface Standard Version 4.0. Retrieved from https:\/\/www.mpi-forum.org\/docs\/mpi-4.0\/mpi40-report.pdf"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVLSI.2015.2478280"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/FPL60245.2023.00029"},{"key":"e_1_3_1_48_2","unstructured":"OmpSs-2. 2024. OmpSs-2 documentation and code. Retrieved March 21 2024 from https:\/\/pm.bsc.es\/ompss-2"},{"key":"e_1_3_1_49_2","unstructured":"OmpSs@FPGA. 2024. Accelerator Integrator Tool (AIT). Retrieved from https:\/\/github.com\/bsc-pm-ompss-at-fpga\/ait"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","unstructured":"Elias Perdomo Alexander Kropotov Francelly Katherine Cano Ladino Syed Zafar Teresa Cervero Xavier Martorell Bofill and Behzad Salami. 2024. Makinote: An FPGA-Based HW\/SW platform for pre-silicon emulation of RISC-V designs. In Proceedings of the 16th Workshop on Rapid Simulation and Performance Evaluation for Design (RAPIDO \u201924). ACM New York NY 29\u201334. 10.1145\/3642921.3642928","DOI":"10.1145\/3642921.3642928"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBAC-PAD63648.2024.00026"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3288431"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3120895.3120909"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3578178.3579341"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-15074-6_3"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC49654.2021.9622807"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM62733.2025.00037"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2019.2907493"},{"key":"e_1_3_1_59_2","unstructured":"Miquel Vidal. 2017. Synchronization\/communication Techniques for OmpSs@FPGA. Master\u2019s thesis. Universitat Polit\u00e8cnica de Catalunya. Retrieved from http:\/\/hdl.handle.net\/2117\/117562"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM.2019.00021"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3431920.3439292"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM48280.2020.00024"},{"key":"e_1_3_1_63_2","unstructured":"Xilinx. 2012. WP380: Xilinx Stacked Silicon Interconnect Technology Delivers Breakthrough FPGA Capacity Bandwidth and Power Efficiency. Retrieved from https:\/\/docs.xilinx.com\/v\/u\/en-US\/wp380_Stacked_Silicon_Interconnect_Technology"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.23919\/ICACT56868.2023.10079513"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2012.03.001"}],"container-title":["ACM Transactions on Reconfigurable Technology and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786349","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T14:03:57Z","timestamp":1779890637000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786349"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,27]]},"references-count":64,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3786349"],"URL":"https:\/\/doi.org\/10.1145\/3786349","relation":{},"ISSN":["1936-7406","1936-7414"],"issn-type":[{"value":"1936-7406","type":"print"},{"value":"1936-7414","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,27]]},"assertion":[{"value":"2025-05-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}