{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T22:44:52Z","timestamp":1777675492410,"version":"3.51.4"},"reference-count":40,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2017,6,29]],"date-time":"2017-06-29T00:00:00Z","timestamp":1498694400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2019,1]]},"abstract":"<jats:p>The gyrokinetic toroidal code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5-D Vlasov\u2013Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P\u2019s multiple levels of parallelism, including internode 2-D domain decomposition and particle decomposition, as well as intranode shared memory partition and vectorization, have enabled pushing the scalability of the PIC method to extreme computational scales. In this article, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) coprocessors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene\/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of ion\u2013temperature\u2013gradient driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputing systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects, and network topologies. These performance comparisons using a realistic discovery-science-capable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide.<\/jats:p>","DOI":"10.1177\/1094342017712059","type":"journal-article","created":{"date-parts":[[2017,7,7]],"date-time":"2017-07-07T10:10:43Z","timestamp":1499422243000},"page":"169-188","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":16,"title":["Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers"],"prefix":"10.1177","volume":"33","author":[{"given":"Bei","family":"Wang","sequence":"first","affiliation":[{"name":"Princeton Institute of Computational Science and Engineering, Princeton University, Princeton, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephane","family":"Ethier","sequence":"additional","affiliation":[{"name":"Princeton Plasma Physics Laboratory, Princeton, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William","family":"Tang","sequence":"additional","affiliation":[{"name":"Princeton Institute of Computational Science and Engineering, Princeton University, Princeton, NJ, USA"},{"name":"Princeton Plasma Physics Laboratory, Princeton, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Khaled Z","family":"Ibrahim","sequence":"additional","affiliation":[{"name":"CRD, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kamesh","family":"Madduri","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University, University Park, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samuel","family":"Williams","sequence":"additional","affiliation":[{"name":"CRD, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leonid","family":"Oliker","sequence":"additional","affiliation":[{"name":"CRD, Lawrence Berkeley National Laboratory, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2017,6,29]]},"reference":[{"issue":"1","key":"bibr1-1094342017712059","first-page":"012001","volume":"78","author":"Adams MF","year":"2007","journal-title":"Journal of Physics: Conference Series"},{"key":"bibr2-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1063\/1.870740"},{"key":"bibr3-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1887\/0750301171"},{"key":"bibr4-1094342017712059","unstructured":"Blue Waters Cray XE6. Available at: http:\/\/www.ncsa.illinois.edu\/enabling\/bluewaters (accessed June 2015)."},{"key":"bibr5-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1063\/1.2751603"},{"key":"bibr6-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1063\/1.873896"},{"key":"bibr7-1094342017712059","unstructured":"Edison Cray XC30. Available at: http:\/\/www.nersc.gov\/systems\/edison-cray-xc30 (accessed June 2015)."},{"key":"bibr8-1094342017712059","volume-title":"Proceeding high performance computing for computational science (VECPAR\u201910) 2010","author":"Ethier S","year":"2010"},{"key":"bibr9-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/16\/1\/001"},{"key":"bibr10-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1147\/rd.521.0105"},{"key":"bibr11-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1177\/1094342013492446"},{"key":"bibr12-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1063\/1.4867379"},{"key":"bibr13-1094342017712059","unstructured":"Joint European Torus (JET) (2013). Available at: http:\/\/www.efda.org\/jet\/ (accessed April 2013)."},{"key":"bibr14-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2007.04.006"},{"key":"bibr15-1094342017712059","unstructured":"Karpusenko V, Vladimirov A (2014) Configuration and benchmarks of peer-to-peer MPI communication over Gigabit Ethernet and InfiniBand in a cluster with Intel Xeon Phi coprocessors. Available at: https:\/\/colfaxresearch.com\/configuration-and-benchmarks-of-peer-to-peer-communication-over-gigabit-ethernet-and-infiniband-in-a-cluster-with-intel-xeon-phi-coprocessors"},{"key":"bibr16-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1088\/0029-5515\/49\/11\/115021"},{"key":"bibr17-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2010.10.013"},{"key":"bibr18-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1016\/0021-9991(87)90080-5"},{"key":"bibr19-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.52.5646"},{"key":"bibr20-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.88.195004"},{"key":"bibr21-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1126\/science.281.5384.1835"},{"key":"bibr22-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1145\/2063384.2063415"},{"issue":"9","key":"bibr23-1094342017712059","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/j.parco.2011.02.001","volume":"37","author":"Madduri K","year":"2011","journal-title":"Parallel Computing"},{"key":"bibr24-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1145\/1654059.1654108"},{"key":"bibr25-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1063\/1.2921792"},{"key":"bibr26-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.105.155001"},{"key":"bibr27-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38750-0_7"},{"key":"bibr28-1094342017712059","unstructured":"Mira IBM BlueGene\/Q. Available at: http:\/\/www.alcf.anl.gov\/mira (accessed June 2015)."},{"key":"bibr29-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.71.2042"},{"key":"bibr30-1094342017712059","unstructured":"Piz Daint Cray XC30. Available at: http:\/\/www.cscs.ch\/computers\/piz_daint\/index.html (accessed June 2015)."},{"key":"bibr31-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4302-5927-5"},{"key":"bibr32-1094342017712059","unstructured":"Stampede. Available at: https:\/\/portal.tacc.utexas.edu\/user-guides\/stampede (accessed June 2015)."},{"key":"bibr33-1094342017712059","first-page":"212","author":"Tang W","year":"2009","journal-title":"PNNL-19404"},{"key":"bibr34-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2014.54"},{"key":"bibr35-1094342017712059","unstructured":"The ITER project (2013). Available at: http:\/\/www.iter.org\/ (accessed April 2013)."},{"key":"bibr36-1094342017712059","unstructured":"Thrust: A template library for CUDA applications (2010). Available at: http:\/\/code.google.com\/p\/thrust\/ (accessed June 2015)."},{"key":"bibr37-1094342017712059","unstructured":"Titan Cray XK7. Available at: https:\/\/www.olcf.ornl.gov\/support\/system-user-guides\/titan-user-guide\/ (accessed June 2015)."},{"key":"bibr38-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1145\/2503210.2503258"},{"key":"bibr39-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1137\/100811805"},{"key":"bibr40-1094342017712059","doi-asserted-by":"publisher","DOI":"10.1137\/120872954"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017712059","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342017712059","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017712059","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:15:39Z","timestamp":1777450539000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342017712059"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,29]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1]]}},"alternative-id":["10.1177\/1094342017712059"],"URL":"https:\/\/doi.org\/10.1177\/1094342017712059","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,29]]}}}