{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:18:07Z","timestamp":1750220287937,"version":"3.41.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"1-4","license":[{"start":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T00:00:00Z","timestamp":1638230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000006","name":"US Office of Naval Research","doi-asserted-by":"crossref","award":["N00014-16-1-2711, N00014-16-1-2104, and N00014-18-1-2022"],"award-info":[{"award-number":["N00014-16-1-2711, N00014-16-1-2104, and N00014-18-1-2022"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NAVSEA\/NEEC","award":["N00174-16-C-0018"],"award-info":[{"award-number":["N00174-16-C-0018"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Syst."],"published-print":{"date-parts":[[2021,11,30]]},"abstract":"<jats:p>\n            In this work, we present\n            <jats:monospace>libHetMP<\/jats:monospace>\n            , an OpenMP runtime for automatically and transparently distributing parallel computation across heterogeneous nodes.\n            <jats:monospace>libHetMP<\/jats:monospace>\n            targets platforms comprising CPUs with different instruction set architectures (ISA) coupled by a high-speed memory interconnect, where cross-ISA binary incompatibility and non-coherent caches require application data be marshaled to be shared across CPUs. Because of this, work distribution decisions must take into account both relative compute performance of asymmetric CPUs and communication overheads.\n            <jats:monospace>libHetMP<\/jats:monospace>\n            drives workload distribution decisions without programmer intervention by measuring performance characteristics during cross-node execution. A novel HetProbe loop iteration scheduler decides if cross-node execution is beneficial and either distributes work according to the relative performance of CPUs when it is or places all work on the set of homogeneous CPUs providing the best performance when it is not. We evaluate\n            <jats:monospace>libHetMP<\/jats:monospace>\n            using compute kernels from several OpenMP benchmark suites and show a geometric mean 41% speedup in execution time across asymmetric CPUs. Because some workloads may showcase irregular behavior among iterations, we extend\n            <jats:monospace>libHetMP<\/jats:monospace>\n            with a second scheduler called HetProbe-I. The evaluation of HetProbe-I shows it can further improve speedup for irregular computation, in some cases up to a 24%, by triggering periodic distribution decisions.\n          <\/jats:p>","DOI":"10.1145\/3505224","type":"journal-article","created":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T21:05:35Z","timestamp":1644008735000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["An OpenMP Runtime for Transparent Work Sharing across Cache-Incoherent Heterogeneous Nodes"],"prefix":"10.1145","volume":"39","author":[{"given":"Robert","family":"Lyerly","sequence":"first","affiliation":[{"name":"Virginia Tech"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4750-5124","authenticated-orcid":false,"given":"Carlos","family":"Bilbao","sequence":"additional","affiliation":[{"name":"Virginia Tech"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Changwoo","family":"Min","sequence":"additional","affiliation":[{"name":"Virginia Tech"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher J.","family":"Rossbach","sequence":"additional","affiliation":[{"name":"University of Texas at Austin and VMware Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Binoy","family":"Ravindran","sequence":"additional","affiliation":[{"name":"Virginia Tech"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,5]]},"reference":[{"unstructured":"2017. PCI Express Base Specification Revision 4.0 Version 1.0. Retrieved from https:\/\/pcisig.com\/specifications\/pciexpress\/.","key":"e_1_3_1_2_2"},{"unstructured":"2018. Summit: A Supercomputer Suited for AI. Retrieved from https:\/\/www.olcf.ornl.gov\/wp-content\/uploads\/2018\/06\/NODE_infographic_FIN.pdf.","key":"e_1_3_1_3_2"},{"unstructured":"AMD. 2020. AMD Infinity Architecture Technology. Retrieved from https:\/\/www.amd.com\/en\/technologies\/infinity-architecture.","key":"e_1_3_1_4_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_5_2","DOI":"10.1109\/2.485843"},{"unstructured":"Anandtech. 2019. Intel Agilex: 10nm FPGAs with PCIe 5.0 DDR5 and CXL. Retrieved from https:\/\/www.anandtech.com\/show\/14149\/intel-agilex-10nm-fpgas-with-pcie-50-ddr5-and-cxl.","key":"e_1_3_1_6_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_7_2","DOI":"10.1177\/109434209100500306"},{"doi-asserted-by":"publisher","key":"e_1_3_1_8_2","DOI":"10.1145\/3037697.3037738"},{"doi-asserted-by":"publisher","key":"e_1_3_1_9_2","DOI":"10.1145\/2741948.2741962"},{"unstructured":"Jeff Barr. 2018. New\u2014EC2 Instances (A1) Powered by Arm-Based AWS Graviton Processors. Retrieved from https:\/\/aws.amazon.com\/blogs\/aws\/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors\/.","key":"e_1_3_1_10_2"},{"key":"e_1_3_1_11_2","volume-title":"Benchmarking Modern Multiprocessors","author":"Bienia Christian","year":"2011","unstructured":"Christian Bienia. 2011. Benchmarking Modern Multiprocessors. Ph.D. Dissertation. Princeton University."},{"doi-asserted-by":"publisher","key":"e_1_3_1_12_2","DOI":"10.1145\/209936.209958"},{"doi-asserted-by":"publisher","key":"e_1_3_1_13_2","DOI":"10.1109\/IPDPS.2012.58"},{"doi-asserted-by":"publisher","key":"e_1_3_1_14_2","DOI":"10.1145\/1094811.1094852"},{"doi-asserted-by":"publisher","key":"e_1_3_1_15_2","DOI":"10.1109\/IISWC.2009.5306797"},{"doi-asserted-by":"publisher","key":"e_1_3_1_16_2","DOI":"10.1145\/1065944.1065950"},{"unstructured":"CCIX Consortium et\u00a0al. 2017. Cache Coherent Interconnect for Accelerators (CCIX). Retreived from http:\/\/www.ccixconsortium.com.","key":"e_1_3_1_17_2"},{"unstructured":"CXL Consortium. 2020. Compute Express Link. Retrieved from https:\/\/www.computeexpresslink.org\/.","key":"e_1_3_1_18_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_19_2","DOI":"10.1145\/3235830.3235833"},{"doi-asserted-by":"publisher","key":"e_1_3_1_20_2","DOI":"10.1145\/2150976.2151004"},{"doi-asserted-by":"publisher","key":"e_1_3_1_21_2","DOI":"10.1145\/2000064.2000108"},{"key":"e_1_3_1_22_2","article-title":"big.LITTLE processing with ARM Cortex-A15 & Cortex-A7","volume":"17","author":"Greenhalgh Peter","year":"2011","unstructured":"Peter Greenhalgh. 2011. big.LITTLE processing with ARM Cortex-A15 & Cortex-A7. ARM White Paper 17 (2011).","journal-title":"ARM White Paper"},{"doi-asserted-by":"publisher","key":"e_1_3_1_23_2","DOI":"10.1007\/978-3-642-19861-8_16"},{"key":"e_1_3_1_24_2","volume-title":"Using MPI: Portable Parallel Programming with the Message-passing Interface","author":"Gropp William","year":"1999","unstructured":"William Gropp, Ewing Lusk, and Anthony Skjellum. 1999. Using MPI: Portable Parallel Programming with the Message-passing Interface. Vol. 1. MIT Press."},{"doi-asserted-by":"publisher","key":"e_1_3_1_25_2","DOI":"10.1109\/SC.2018.00064"},{"key":"e_1_3_1_26_2","article-title":"Extending OpenMP to clusters","author":"Hoeflinger Jay P.","year":"2006","unstructured":"Jay P. Hoeflinger. 2006. Extending OpenMP to clusters. White Paper, Intel Corporation.","journal-title":"White Paper, Intel Corporation"},{"doi-asserted-by":"publisher","key":"e_1_3_1_27_2","DOI":"10.1145\/2854038.2854047"},{"doi-asserted-by":"publisher","key":"e_1_3_1_28_2","DOI":"10.5555\/871085"},{"key":"e_1_3_1_29_2","volume-title":"The OpenCL Specification","author":"Group Khronos OpenCL Working","year":"2018","unstructured":"Khronos OpenCL Working Group. 2018. The OpenCL Specification. Technical Report. https:\/\/www.khronos.org\/registry\/OpenCL\/specs\/2.2\/pdf\/OpenCL_API.pdf."},{"doi-asserted-by":"publisher","key":"e_1_3_1_30_2","DOI":"10.1145\/2908080.2908094"},{"unstructured":"Sang-Hoon Kim Robert Lyerly and Pierre Olivier. 2017. Popcorn Linux: Compiler Operating System and Virtualization Support for Application\/Thread Migration in Heterogeneous ISA Environments. Retrieved from http:\/\/www.linuxplumbersconf.org\/2017\/ocw\/proposals\/4719.html.","key":"e_1_3_1_31_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_32_2","DOI":"10.1145\/2464996.2465007"},{"unstructured":"Akhilesh Kumar. 2017. The New Intel Xeon Processor Scalable Family (Formerly Skylake-SP). Retreived from https:\/\/www.hotchips.org\/wp-content\/uploads\/hc_archives\/hc29\/HC29.22-Tuesday-Pub\/HC29.22.90-Server-Pub\/HC29.22.930-Xeon-Skylake-sp-Kumar-Intel.pdf.","key":"e_1_3_1_33_2"},{"key":"e_1_3_1_34_2","article-title":"The next generation AMD enterprise server product architecture","volume":"29","author":"Lepak Keven","year":"2017","unstructured":"Keven Lepak, Gerry Talbot, Sean White, Noah Beck, S. Naffziger, et\u00a0al. 2017. The next generation AMD enterprise server product architecture. IEEE Hot Chips 29 (2017).","journal-title":"IEEE Hot Chips"},{"doi-asserted-by":"publisher","key":"e_1_3_1_35_2","DOI":"10.1145\/2541940.2541975"},{"doi-asserted-by":"publisher","key":"e_1_3_1_36_2","DOI":"10.1145\/1669112.1669121"},{"doi-asserted-by":"publisher","key":"e_1_3_1_37_2","DOI":"10.1145\/3303084.3309495"},{"doi-asserted-by":"publisher","key":"e_1_3_1_38_2","DOI":"10.1145\/3423211.3425679"},{"doi-asserted-by":"publisher","key":"e_1_3_1_39_2","DOI":"10.1109\/CLUSTR.2004.1392625"},{"doi-asserted-by":"publisher","key":"e_1_3_1_40_2","DOI":"10.1007\/978-3-642-30961-8_17"},{"key":"e_1_3_1_41_2","first-page":"291","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201915)","author":"Nelson Jacob","year":"2015","unstructured":"Jacob Nelson, Brandon Holt, Brandon Myers, Preston Briggs, Luis Ceze, Simon Kahan, and Mark Oskin. 2015. Latency-Tolerant software distributed shared memory. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201915). USENIX Association, Berkeley, CA, 291\u2013305. https:\/\/www.usenix.org\/conference\/atc15\/technical-session\/presentation\/nelson."},{"unstructured":"NVidia. 2020. NVLink. Retrieved from https:\/\/www.nvidia.com\/en-us\/data-center\/nvlink\/.","key":"e_1_3_1_42_2"},{"unstructured":"OpenCAPI Consortium. 2020. OpenCAPI Consortium. Retrieved from https:\/\/opencapi.org\/.","key":"e_1_3_1_43_2"},{"key":"e_1_3_1_44_2","volume-title":"OpenMP Application Program Interface v5.0","author":"Board OpenMP Architecture Review","year":"2018","unstructured":"OpenMP Architecture Review Board. 2018. OpenMP Application Program Interface v5.0. Technical Report. OpenMP Architecture Review Board. Retrieved from https:\/\/www.openmp.org\/wp-content\/uploads\/OpenMP-API-Specification-5.0.pdf."},{"key":"e_1_3_1_45_2","first-page":"7","volume-title":"Proceedings of the USENIX Conference on Power-Aware Computing and Systems (HotPower\u201912)","author":"Petrucci Vinicius","year":"2012","unstructured":"Vinicius Petrucci, Orlando Loques, and Daniel Moss\u00e9. 2012. Lucky scheduling for energy-efficient heterogeneous multi-core systems. In Proceedings of the USENIX Conference on Power-Aware Computing and Systems (HotPower\u201912). USENIX Association, Berkeley, CA, 7\u20137. http:\/\/dl.acm.org\/citation.cfm?id=2387869.2387876."},{"unstructured":"The Next Platform. 2018. Next-Generation ThunderX2 ARM Targets Skylake Xeons. Retrieved from https:\/\/www.nextplatform.com\/2016\/06\/03\/next-generation-thunderx2-arm-targets-skylake-xeons\/.","key":"e_1_3_1_46_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_47_2","DOI":"10.1145\/2996868"},{"unstructured":"Qualcomm. 2019. Qualcomm Snapdragon 855 Mobile Platform. Retrieved from https:\/\/www.qualcomm.com\/media\/documents\/files\/snapdragon-855-mobile-platform-product-brief.pdf.","key":"e_1_3_1_48_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_49_2","DOI":"10.1145\/1839667.1839671"},{"doi-asserted-by":"publisher","key":"e_1_3_1_50_2","DOI":"10.1145\/3168390.3168401"},{"doi-asserted-by":"publisher","key":"e_1_3_1_51_2","DOI":"10.1109\/TPDS.2014.2365192"},{"doi-asserted-by":"publisher","key":"e_1_3_1_52_2","DOI":"10.1109\/IISWC.2011.6114174"},{"doi-asserted-by":"publisher","key":"e_1_3_1_53_2","DOI":"10.1145\/3136063"},{"issue":"3","key":"e_1_3_1_54_2","first-page":"202","article-title":"The free lunch is over: A fundamental turn toward concurrency in software","volume":"30","author":"Sutter Herb","year":"2005","unstructured":"Herb Sutter. 2005. The free lunch is over: A fundamental turn toward concurrency in software. Dr. Dobb\u2019s J. 30, 3 (2005), 202\u2013210.","journal-title":"Dr. Dobb\u2019s J."},{"unstructured":"Herb Sutter. 2012. Welcome to the Jungle. Retrieved from https:\/\/herbsutter.com\/welcome-to-the-jungle\/.","key":"e_1_3_1_55_2"},{"doi-asserted-by":"publisher","key":"e_1_3_1_56_2","DOI":"10.1145\/2678373.2665692"},{"doi-asserted-by":"publisher","key":"e_1_3_1_57_2","DOI":"10.1145\/197320.197402"}],"container-title":["ACM Transactions on Computer Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3505224","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3505224","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:26Z","timestamp":1750188686000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3505224"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,30]]},"references-count":56,"journal-issue":{"issue":"1-4","published-print":{"date-parts":[[2021,11,30]]}},"alternative-id":["10.1145\/3505224"],"URL":"https:\/\/doi.org\/10.1145\/3505224","relation":{},"ISSN":["0734-2071","1557-7333"],"issn-type":[{"type":"print","value":"0734-2071"},{"type":"electronic","value":"1557-7333"}],"subject":[],"published":{"date-parts":[[2021,11,30]]},"assertion":[{"value":"2021-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}