{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,20]],"date-time":"2025-12-20T22:32:15Z","timestamp":1766269935974,"version":"3.41.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T00:00:00Z","timestamp":1576540800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2019,12,31]]},"abstract":"<jats:p>In-silico brain simulations are the de-facto tools computational neuroscientists use to understand large-scale and complex brain-function dynamics. Current brain simulators do not scale efficiently enough to large-scale problem sizes (e.g., &gt;100,000 neurons) when simulating biophysically complex neuron models. The goal of this work is to explore the use of true multi-GPU acceleration through NVIDIA\u2019s GPUDirect technology on computationally challenging brain models and to assess their scalability. The brain model used is a state-of-the-art, extended Hodgkin-Huxley, biophysically meaningful, three-compartmental model of the inferior-olivary nucleus. The Hodgkin-Huxley model is the most widely adopted conductance-based neuron representation, and thus the results from simulating this representative workload are relevant for many other brain experiments. Not only the actual network-simulation times but also the network-setup times were taken into account when designing and benchmarking the multi-GPU version, an aspect often ignored in similar previous work. Network sizes varying from 65K to 2M cells, with 10 and 1,000 synapses per neuron were executed on 8, 16, 24, and 32 GPUs. Without loss of generality, simulations were run for 100 ms of biological time. Findings indicate that communication overheads do not dominate overall execution while scaling the network size up is computationally tractable. This scalable design proves that large-network simulations of complex neural models are possible using a multi-GPU design with GPUDirect.<\/jats:p>","DOI":"10.1145\/3371235","type":"journal-article","created":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T13:21:11Z","timestamp":1576675271000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Exploring Complex Brain-Simulation Workloads on Multi-GPU Deployments"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0370-4655","authenticated-orcid":false,"given":"Michiel A. van der","family":"Vlag","sequence":"first","affiliation":[{"name":"Erasmus Medical Centre - Department of Neuroscience, Rotterdam, The Netherlands"}]},{"given":"Georgios","family":"Smaragdos","sequence":"additional","affiliation":[{"name":"Erasmus Medical Centre - Department of Neuroscience, Rotterdam, The Netherlands"}]},{"given":"Zaid","family":"Al-Ars","sequence":"additional","affiliation":[{"name":"Delft University of Technology - Quantum 8 Computing Engineering, Rotterdam, The Netherlands"}]},{"given":"Christos","family":"Strydis","sequence":"additional","affiliation":[{"name":"Erasmus Medical Centre - Department of Neuroscience, Rotterdam, The Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2019,12,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2018.10.062"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN\u201918)","author":"Kashyap Hirak J","year":"2018","unstructured":"Ting-shuo Chou, Hirak J Kashyap , Jinwei Xing , Stanislav Listopad , Emily L Rounds , Michael Beyeler , Nikil Dutt , and Jeffrey L Krichmar . 2018 . CARLsim 4 : An open source library for large scale, biologically detailed spiking neural network simulation using heterogeneous clusters . In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN\u201918) . 1158--1165. Ting-shuo Chou, Hirak J Kashyap, Jinwei Xing, Stanislav Listopad, Emily L Rounds, Michael Beyeler, Nikil Dutt, and Jeffrey L Krichmar. 2018. CARLsim 4 : An open source library for large scale, biologically detailed spiking neural network simulation using heterogeneous clusters. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN\u201918). 1158--1165."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/99.660313"},{"key":"e_1_2_1_4_1","volume-title":"PyNN: A common interface for neuronal network simulators. Front. Neuroinform. 2 (Jan","author":"Davison Andrew P.","year":"2008","unstructured":"Andrew P. Davison . 2008. PyNN: A common interface for neuronal network simulators. Front. Neuroinform. 2 (Jan . 2008 ), 1--10. DOI:https:\/\/doi.org\/10.3389\/neuro.11.011.2008 10.3389\/neuro.11.011.2008 Andrew P. Davison. 2008. PyNN: A common interface for neuronal network simulators. Front. Neuroinform. 2 (Jan. 2008), 1--10. DOI:https:\/\/doi.org\/10.3389\/neuro.11.011.2008"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1002814"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3011"},{"volume-title":"Gpu-based Simulation of Brain Neuron Models. Master\u2019s thesis","author":"Nguyen Hoang Du","key":"e_1_2_1_7_1","unstructured":"Hoang Du Nguyen . 2013. Gpu-based Simulation of Brain Neuron Models. Master\u2019s thesis . Technical University of Delft . Hoang Du Nguyen. 2013. Gpu-based Simulation of Brain Neuron Models. Master\u2019s thesis. Technical University of Delft."},{"volume-title":"Spiking Neuron Models: An Introduction","author":"Gerstner Wulfram","key":"e_1_2_1_8_1","unstructured":"Wulfram Gerstner and Werner Kistler . 2002. Spiking Neuron Models: An Introduction . Cambridge University Press , New York, NY . Wulfram Gerstner and Werner Kistler. 2002. Spiking Neuron Models: An Introduction. Cambridge University Press, New York, NY."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.4249\/scholarpedia.1430"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.3389\/neuro.11.005.2008"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2011.00049"},{"key":"e_1_2_1_12_1","first-page":"1","article-title":"The NEURON simulation environment, neural computation","volume":"9","author":"Hines M. L.","year":"1997","unstructured":"M. L. Hines and N. T. Carnevale . 1997 . The NEURON simulation environment, neural computation . Pubmed 9 , 6 (1997), 1 -- 26 . M. L. Hines and N. T. Carnevale. 1997. The NEURON simulation environment, neural computation. Pubmed 9, 6 (1997), 1--26.","journal-title":"Pubmed"},{"key":"e_1_2_1_13_1","volume-title":"Harris","author":"Hoang Roger V.","year":"2013","unstructured":"Roger V. Hoang , Devyani Tanna , Laurence C. Jayet Bray , Sergiu M. Dascalu , and Frederick C . Harris . 2013 . A novel CPU\/GPU simulation environment for large-scale biologically realistic neural modeling. Front. Neuroinform . 7 (Oct. 2013), 19. DOI:https:\/\/doi.org\/10.3389\/fninf.2013.00019 10.3389\/fninf.2013.00019 Roger V. Hoang, Devyani Tanna, Laurence C. Jayet Bray, Sergiu M. Dascalu, and Frederick C. Harris. 2013. A novel CPU\/GPU simulation environment for large-scale biologically realistic neural modeling. Front. Neuroinform. 7 (Oct. 2013), 19. DOI:https:\/\/doi.org\/10.3389\/fninf.2013.00019"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.2004.832719"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2018.00941"},{"key":"e_1_2_1_16_1","unstructured":"Jiri Kraus. 2013. An Introduction to CUDA-Aware MPI. Retrieved from https:\/\/devblogs.nvidia.com\/introduction-cuda-aware-mpi\/.  Jiri Kraus. 2013. An Introduction to CUDA-Aware MPI. Retrieved from https:\/\/devblogs.nvidia.com\/introduction-cuda-aware-mpi\/."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2019.00063"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3233\/JAD-2001-3201"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 9th International Conference on Neural Information Processing Systems (NIPS\u201996)","author":"Maass Wolfgang","year":"1996","unstructured":"Wolfgang Maass . 1996 . Noisy spiking neurons with temporal coding have more computational power than sigmoidal neurons . In Proceedings of the 9th International Conference on Neural Information Processing Systems (NIPS\u201996) . MIT Press, Cambridge, MA, 211--217. http:\/\/dl.acm.org\/citation.cfm?id&equals;2998981.2999011 Wolfgang Maass. 1996. Noisy spiking neurons with temporal coding have more computational power than sigmoidal neurons. In Proceedings of the 9th International Conference on Neural Information Processing Systems (NIPS\u201996). MIT Press, Cambridge, MA, 211--217. http:\/\/dl.acm.org\/citation.cfm?id&equals;2998981.2999011"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(97)00011-7"},{"volume-title":"Methods in Neuronal Modeling","author":"Mascagni Micheal V.","key":"e_1_2_1_21_1","unstructured":"Micheal V. Mascagni . 1989. Numerical methods for neuronal modeling . In Methods in Neuronal Modeling , Christof Koch and Idan Segev (Eds.). MIT Press , Cambridge, MA , 439--484. http:\/\/dl.acm.org\/citation.cfm?id&equals;94605.94628 Micheal V. Mascagni. 1989. Numerical methods for neuronal modeling. In Methods in Neuronal Modeling, Christof Koch and Idan Segev (Eds.). MIT Press, Cambridge, MA, 439--484. http:\/\/dl.acm.org\/citation.cfm?id&equals;94605.94628"},{"volume-title":"flexHH: A Flexible Hardware Library for Hodgkin-Huxley-based Neural Simulations. Master\u2019s thesis","author":"Miedema Rene","key":"e_1_2_1_22_1","unstructured":"Rene Miedema . 2019. flexHH: A Flexible Hardware Library for Hodgkin-Huxley-based Neural Simulations. Master\u2019s thesis . Technical University of Delft . Rene Miedema. 2019. flexHH: A Flexible Hardware Library for Hodgkin-Huxley-based Neural Simulations. Master\u2019s thesis. Technical University of Delft."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2276056"},{"key":"e_1_2_1_24_1","volume-title":"De Zeeuw","author":"Negrello Mario","year":"2019","unstructured":"Mario Negrello , Pascal Warnaar , Vincenzo Romano , Cullen B. Owens , Sander Lindeman , Elisabetta Iavarone , Jochen K. Spanke , Laurens W. J. Bosman , and Chris I . De Zeeuw . 2019 . Quasiperiodic rhythms of the inferior olive. PLOS Comput. Biol . 15, 5 (05 2019), 1--41. DOI:https:\/\/doi.org\/10.1371\/journal.pcbi.1006475 10.1371\/journal.pcbi.1006475 Mario Negrello, Pascal Warnaar, Vincenzo Romano, Cullen B. Owens, Sander Lindeman, Elisabetta Iavarone, Jochen K. Spanke, Laurens W. J. Bosman, and Chris I. De Zeeuw. 2019. Quasiperiodic rhythms of the inferior olive. PLOS Comput. Biol. 15, 5 (05 2019), 1--41. DOI:https:\/\/doi.org\/10.1371\/journal.pcbi.1006475"},{"key":"e_1_2_1_25_1","unstructured":"NVIDIA. 2010. GPUdirect. Retrieved from https:\/\/developer.nvidia.com\/gpudirect.  NVIDIA. 2010. GPUdirect. Retrieved from https:\/\/developer.nvidia.com\/gpudirect."},{"key":"e_1_2_1_26_1","unstructured":"Performance Portability. 2018. Measuring Roofline Quantities on NVIDIA GPUs. Retrieved from http:\/\/performanceportability.org\/perfport\/measurements\/gpu\/.  Performance Portability. 2018. Measuring Roofline Quantities on NVIDIA GPUs. Retrieved from http:\/\/performanceportability.org\/perfport\/measurements\/gpu\/."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1006\/jcph.1995.1039"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/wcms.1121"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aa7fc5"},{"key":"e_1_2_1_30_1","series-title":"Lecture Notes in Computer Science","volume-title":"Supercomputing","author":"Smaragdos Georgios","unstructured":"Georgios Smaragdos , Craig Davies , Christos Strydis , Ioannis Sourdis , Catalin Ciobanu , Oskar Mencer , and Chris De Zeeuw . 2014. Real-time olivary neuron simulations on dataflow computing machines . In Supercomputing . Lecture Notes in Computer Science , Vol. 8488 . DOI:https:\/\/doi.org\/10.1007\/978-3-319-07518-1_34 10.1007\/978-3-319-07518-1_34 Georgios Smaragdos, Craig Davies, Christos Strydis, Ioannis Sourdis, Catalin Ciobanu, Oskar Mencer, and Chris De Zeeuw. 2014. Real-time olivary neuron simulations on dataflow computing machines. In Supercomputing. Lecture Notes in Computer Science, Vol. 8488. DOI:https:\/\/doi.org\/10.1007\/978-3-319-07518-1_34"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Marcel Stimberg Dan F. M. Goodman Victor Benichoux and Romain Brette. 2013. Brian 2 - the second coming: Spiking neural network simulation in Python with code generation. In BMC Neuroscience. BioMed Central 1471--2202.  Marcel Stimberg Dan F. M. Goodman Victor Benichoux and Romain Brette. 2013. Brian 2 - the second coming: Spiking neural network simulation in Python with code generation. In BMC Neuroscience. BioMed Central 1471--2202.","DOI":"10.1186\/1471-2202-14-S1-P38"},{"key":"e_1_2_1_32_1","unstructured":"Surfsara. 2016. Cartesius: the Dutch supercomputer. Retrieved from https:\/\/userinfo.surfsara.nl\/systems\/cartesius.  Surfsara. 2016. Cartesius: the Dutch supercomputer. Retrieved from https:\/\/userinfo.surfsara.nl\/systems\/cartesius."},{"key":"e_1_2_1_33_1","volume-title":"Retrieved","author":"Technologies Mellanox","year":"2013","unstructured":"Mellanox Technologies . 2013 . Accelerating High Performance Computing with GPUDirect RDMA . Retrieved October 14, 2019 from http:\/\/on-demand.gputechconf.com\/gtc\/2013\/webinar\/gtc-express-gpudirect-rdma.pdf. Mellanox Technologies. 2013. Accelerating High Performance Computing with GPUDirect RDMA. Retrieved October 14, 2019 from http:\/\/on-demand.gputechconf.com\/gtc\/2013\/webinar\/gtc-express-gpudirect-rdma.pdf."},{"key":"e_1_2_1_34_1","first-page":"26","article-title":"Conditioned climbing fiber responses in cerebellar cortex and nuclei. Neurosci","volume":"688","author":"Brinke M. M.","year":"2019","unstructured":"M. M. ten Brinke , H. J. Boele , and C. I. De Zeeuw . 2019 . Conditioned climbing fiber responses in cerebellar cortex and nuclei. Neurosci . Lett. 688 (2019), 26 -- 36 . DOI:https:\/\/doi.org\/10.1016\/j.neulet.2018.04.035 The Cerebellum in Health and Disease. 10.1016\/j.neulet.2018.04.035 M. M. ten Brinke, H. J. Boele, and C. I. De Zeeuw. 2019. Conditioned climbing fiber responses in cerebellar cortex and nuclei. Neurosci. Lett. 688 (2019), 26--36. DOI:https:\/\/doi.org\/10.1016\/j.neulet.2018.04.035 The Cerebellum in Health and Disease.","journal-title":"Lett."},{"key":"e_1_2_1_35_1","volume-title":"Hamker","author":"Vitay Julien","year":"2015","unstructured":"Julien Vitay , Helge \u00dc. Dinkelbach , and Fred H . Hamker . 2015 . ANNarchy: A code generation approach to neural simulations on parallel hardware. Front. Neuroinform . 9 (Jul. 2015), 1--20. DOI:https:\/\/doi.org\/10.3389\/fninf.2015.00019 10.3389\/fninf.2015.00019 Julien Vitay, Helge \u00dc. Dinkelbach, and Fred H. Hamker. 2015. ANNarchy: A code generation approach to neural simulations on parallel hardware. Front. Neuroinform. 9 (Jul. 2015), 1--20. DOI:https:\/\/doi.org\/10.3389\/fninf.2015.00019"},{"key":"e_1_2_1_36_1","volume-title":"Yosef Yarom, Mario Negrello, Ben Torben-Nielsen, and Marylka Yoe Uusisaari.","author":"Vrieler Nora","year":"2019","unstructured":"Nora Vrieler , Sebastian Loyola , Yasmin Yarden-Rabinowitz , Jesse Hoogendorp , Nikolay Medvedev , Tycho M. Hoogland , Chris I. De Zeeuw , Erik De Schutter , Yosef Yarom, Mario Negrello, Ben Torben-Nielsen, and Marylka Yoe Uusisaari. 2019 . Variability and directionality of inferior olive neuron dendrites revealed by detailed 3D characterization of an extensive morphological library. Brain Struct. Funct . 224, 4 (01 May 2019), 1677--1695. DOI:https:\/\/doi.org\/10.1007\/s00429-019-01859-z 10.1007\/s00429-019-01859-z Nora Vrieler, Sebastian Loyola, Yasmin Yarden-Rabinowitz, Jesse Hoogendorp, Nikolay Medvedev, Tycho M. Hoogland, Chris I. De Zeeuw, Erik De Schutter, Yosef Yarom, Mario Negrello, Ben Torben-Nielsen, and Marylka Yoe Uusisaari. 2019. Variability and directionality of inferior olive neuron dendrites revealed by detailed 3D characterization of an extensive morphological library. Brain Struct. Funct. 224, 4 (01 May 2019), 1677--1695. DOI:https:\/\/doi.org\/10.1007\/s00429-019-01859-z"},{"key":"e_1_2_1_37_1","first-page":"18854","article-title":"GeNN: A code generation framework for accelerated brain simulations. Sci","volume":"6","author":"Yavuz Esin","year":"2016","unstructured":"Esin Yavuz , James Turner , and Thomas Nowotny . 2016 . GeNN: A code generation framework for accelerated brain simulations. Sci . Rep. 6 , 1 (2016), 18854 . DOI:https:\/\/doi.org\/10.1038\/srep18854 10.1038\/srep18854 Esin Yavuz, James Turner, and Thomas Nowotny. 2016. GeNN: A code generation framework for accelerated brain simulations. Sci. Rep. 6, 1 (2016), 18854. DOI:https:\/\/doi.org\/10.1038\/srep18854","journal-title":"Rep."}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3371235","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3371235","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:04Z","timestamp":1750204384000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3371235"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,17]]},"references-count":37,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,12,31]]}},"alternative-id":["10.1145\/3371235"],"URL":"https:\/\/doi.org\/10.1145\/3371235","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"type":"print","value":"1544-3566"},{"type":"electronic","value":"1544-3973"}],"subject":[],"published":{"date-parts":[[2019,12,17]]},"assertion":[{"value":"2019-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}