{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T17:10:13Z","timestamp":1759165813068,"version":"3.44.0"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T00:00:00Z","timestamp":1786665600000},"content-version":"vor","delay-in-days":319,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100008902","name":"Los Alamos National Laboratory","doi-asserted-by":"crossref","award":["LA-UR-24-20385"],"award-info":[{"award-number":["LA-UR-24-20385"]}],"id":[{"id":"10.13039\/100008902","id-type":"DOI","asserted-by":"crossref"}]},{"name":"U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Early Career Research Program, DyGenAI project"},{"name":"SEA-CROGS project in the MMICCs program"},{"name":"National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy\u2019s National Nuclear Security Administration","award":["DE-NA0003525"],"award-info":[{"award-number":["DE-NA0003525"]}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["INST 218\/78-1 FUGG"],"award-info":[{"award-number":["INST 218\/78-1 FUGG"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"crossref"}]},{"name":"IFI programme of the German Academic Exchange Service"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Math. Softw."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>TorchBraid is a high-performance implementation of layer-parallel training for deep neural networks (DNNs) supporting MPI-based parallelism and GPU acceleration. Layer-parallel training has been developed to overcome the serialization inherent in forward and backward propagation of DNNs that limits utilization of computational resources in the strong scaling limit. To achieve this, TorchBraid integrates the PyTorch neural network framework with the state-of-the-art XBraid time-parallel library. This article presents the use and performance of TorchBraid, in addition to solutions for overcoming the algorithmic challenges inherent in combining automatic differentiation with layer-parallel. Results are presented with and without GPU acceleration for the Tiny ImageNet and MNIST image classification data sets, as well as recurrent neural networks. Overall, TorchBraid enables fast training of DNNs, both in a strong and weak scaling context.<\/jats:p>\n          <jats:p>In addition to the TorchBraid software, several new advances in applying layer-parallel algorithms are detailed. Integration of layer-parallel with data-parallel algorithms is presented for the first time, showing the computational advantages of the combination. Standard deep learning techniques, like batch-normalization, are developed for layer-parallel training. Finally, a new approach combining layer-parallel with spatial coarsening in order to accelerate training for 3D image classification shows roughly a 10\u00d7 speedup over serial execution.<\/jats:p>","DOI":"10.1145\/3759244","type":"journal-article","created":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T18:39:51Z","timestamp":1755196791000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TorchBraid: High-Performance Layer-Parallel Training of Deep Neural Networks with MPI and GPU Acceleration"],"prefix":"10.1145","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3833-9598","authenticated-orcid":false,"given":"Eric C.","family":"Cyr","sequence":"first","affiliation":[{"name":"Sandia National Laboratories, Albuquerque, New Mexico, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7711-7894","authenticated-orcid":false,"given":"Jens","family":"Hahne","sequence":"additional","affiliation":[{"name":"Bergische Universit\u00e4t Wuppertal, Wuppertal, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5224-9113","authenticated-orcid":false,"given":"Nicholas S.","family":"Moore","sequence":"additional","affiliation":[{"name":"West Texas A&amp;M University, Canyon, Texas, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1076-9206","authenticated-orcid":false,"given":"Jacob B.","family":"Schroder","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics, University of New Mexico, Albuquerque, New Mexico, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0283-4928","authenticated-orcid":false,"given":"Ben S.","family":"Southworth","sequence":"additional","affiliation":[{"name":"Los Alamos National Laboratory, Los Alamos, New Mexico, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7786-5753","authenticated-orcid":false,"given":"David A.","family":"Vargas","sequence":"additional","affiliation":[{"name":"Sandia National Laboratories, Albuquerque, New Mexico, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,29]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved from https:\/\/www.tensorflow.org\/Software available from tensorflow.org"},{"issue":"153","key":"e_1_3_2_3_2","first-page":"1","article-title":"Automatic differentiation in machine learning: A survey","volume":"18","author":"Baydin Atilim Gunes","year":"2018","unstructured":"Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: A survey. Journal of Machine Learning Research 18, 153 (2018), 1\u201343.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2010.118"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3320060"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1137\/16M1080173"},{"key":"e_1_3_2_7_2","unstructured":"James Bradbury Roy Frostig Peter Hawkins Matthew James Johnson Chris Leary Dougal Maclaurin George Necula Adam Paszke Jake VanderPlas Skye Wanderman-Milne et al. 2018. JAX: Composable Transformations of Python+NumPy Programs. Retrieved from http:\/\/github.com\/google\/jax"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611970753","volume-title":"Multigrid Techniques: 1984 Guide with Applications to Fluid Dynamics","author":"Brandt Achi","year":"2011","unstructured":"Achi Brandt and Oren E. Livne. 2011. Multigrid Techniques: 1984 Guide with Applications to Fluid Dynamics, Revised Edition. SIAM, Philadelphia, PA."},{"key":"e_1_3_2_9_2","volume-title":"Algebraic Multigrid (AMG) for Automatic Multigrid Solution with Application to Geodetic Computations","author":"Brandt A.","year":"1982","unstructured":"A. Brandt, S. F. McCormick, and J. W. Ruge. 1982. Algebraic Multigrid (AMG) for Automatic Multigrid Solution with Application to Geodetic Computations. Technical Report. Institute for Computational Studies, Colorado State University."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/347185"},{"key":"e_1_3_2_11_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11668"},{"key":"e_1_3_2_13_2","volume-title":"Advances in Neural Information Processing Systems","author":"Chen Ricky T. Q.","year":"2018","unstructured":"Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. 2018. Neural ordinary differential equations. In Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/file\/69386f6bb1dfed68692a24c8686939b9-Paper.pdf"},{"key":"e_1_3_2_14_2","unstructured":"Kyunghyun Cho Bart Van Merri\u00ebnboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078. Retrieved from https:\/\/arxiv.org\/abs\/1406.1078"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACSOS-C52956.2021.00029"},{"issue":"3","key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1109\/74.250128","article-title":"The fast multipole method for the wave equation: A pedestrian prescription","volume":"35","author":"Coifman Ronald","year":"1993","unstructured":"Ronald Coifman, Vladimir Rokhlin, and Stephen Wandzura. 1993. The fast multipole method for the wave equation: A pedestrian prescription. IEEE Antennas and Propagation Magazine 35, 3 (1993), 7\u201312.","journal-title":"IEEE Antennas and Propagation Magazine"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2021.3083216"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2005.03.010"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2007.09.005"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.advwatres.2011.04.013"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1137\/21M1390773"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1137\/0719025"},{"key":"e_1_3_2_23_2","first-page":"23158","article-title":"A guide through the zoo of biased SGD","volume":"36","author":"Demidovich Yury","year":"2023","unstructured":"Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richt\u00e1rik. 2023. A guide through the zoo of biased SGD. Advances in Neural Information Processing Systems 36 (2023), 23158\u201323171.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1137\/16M1074096"},{"key":"e_1_3_2_25_2","unstructured":"Zhengxiao Du Yujie Qian Xiao Liu Ming Ding Jiezhong Qiu Zhilin Yang and Jie Tang. 2021. All NLP tasks are generation tasks: A general pretraining framework. arXiv:2103.10360. Retrieved from Retrieved from https:\/\/arxiv.org\/abs\/2103.10360"},{"key":"e_1_3_2_26_2","volume-title":"Advances in Neural Information Processing Systems","author":"Dupont Emilien","year":"2019","unstructured":"Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. 2019. Augmented neural ODEs. In Advances in Neural Information Processing Systems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d\u2019Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/file\/21be9a4bd4f81549a9d1d241981cec3c-Paper.pdf"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40304-017-0103-z"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1137\/0804022"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/8.144597"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1137\/130944230"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00791-017-0283-9"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00344251"},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/978-3-319-23321-5_3","volume-title":"Multiple Shooting and Time Domain Decomposition","author":"Gander M. J.","year":"2015","unstructured":"M. J. Gander. 2015. 50 years of time parallel time integration. In Multiple Shooting and Time Domain Decomposition. T. Carraro, M. Geiger, S. K\u00f6rkel, and R. Rannacher (Eds.), Springer, 69\u2013114."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/0021-9991(87)90140-9"},{"issue":"1","key":"e_1_3_2_36_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1145\/347837.347846","article-title":"Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation","volume":"26","author":"Griewank Andreas","year":"2000","unstructured":"Andreas Griewank and Andrea Walther. 2000. Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software 26, 1 (2000), 19\u201345.","journal-title":"ACM Transactions on Mathematical Software"},{"issue":"1","key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1137\/19M1247620","article-title":"Layer-parallel training of deep residual neural networks","volume":"2","author":"G\u00fcnther Stefanie","year":"2020","unstructured":"Stefanie G\u00fcnther, Lars Ruthotto, Jacob B. Schroder, Eric C. Cyr, and Nicolas R. Gauger. 2020. Layer-parallel training of deep residual neural networks. SIAM Journal on Mathematics of Data Science 2, 1 (2020), 1\u201323.","journal-title":"SIAM Journal on Mathematics of Data Science"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/aa9a90"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1080\/00295639.2020.1747263"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2649-2"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_42_2","first-page":"630","volume-title":"14th European Conference on Computer Vision (ECCV \u201916)","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In 14th European Conference on Computer Vision (ECCV \u201916). Springer, 630\u2013645."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1137\/19M1238812"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.79.8.2554"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1137\/17M1144982"},{"key":"e_1_3_2_47_2","article-title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism","volume":"32","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, Vol. 32.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_48_2","first-page":"448","volume-title":"International Conference on Machine Learning. PMLR","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. PMLR, 448\u2013456."},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"678158","DOI":"10.3389\/fncom.2021.678158","article-title":"Gated recurrent units viewed through the lens of continuous time dynamical systems","volume":"15","author":"Jordan Ian D.","year":"2021","unstructured":"Ian D. Jordan, Piotr Aleksander Sok\u00f3\u0142, and Il Memming Park. 2021. Gated recurrent units viewed through the lens of continuous time dynamical systems. Frontiers in Computational Neuroscience 15 (2021), 678158.","journal-title":"Frontiers in Computational Neuroscience"},{"key":"e_1_3_2_50_2","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https:\/\/arxiv.org\/abs\/1412.6980"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"S254","DOI":"10.1137\/21M1434076","article-title":"Globally convergent multilevel training of deep residual networks","volume":"45","author":"Kopanic\u00e1kov\u00e1 Alena","year":"2022","unstructured":"Alena Kopanic\u00e1kov\u00e1 and Rolf Krause. 2022. Globally convergent multilevel training of deep residual networks. SIAM Journal on Scientific Computing 45 (2022), S254\u2013S280.","journal-title":"SIAM Journal on Scientific Computing"},{"issue":"7","key":"e_1_3_2_52_2","first-page":"3","article-title":"Tiny ImageNet visual recognition challenge","volume":"7","author":"Le Ya","year":"2015","unstructured":"Ya Le and Xuan Yang. 2015. Tiny ImageNet visual recognition challenge. CS 231n 7, 7 (2015), 3.","journal-title":"CS 231n"},{"key":"e_1_3_2_53_2","first-page":"1","volume-title":"Power and Energy Society General Meeting (PESGM)","author":"Lecouvez M.","year":"2016","unstructured":"M. Lecouvez, R. D. Falgout, C. S. Woodward, and P. Top. 2016. A parallel multigrid reduction in time method for power systems. In Power and Energy Society General Meeting (PESGM), 1\u20135."},{"key":"e_1_3_2_54_2","first-page":"3361","article-title":"Convolutional networks for images, speech, and time series","author":"LeCun Yann","year":"1995","unstructured":"Yann LeCun and Yoshua Bengio. 1995. Convolutional networks for images, speech, and time series. In The Handbook of Brain Theory and Neural Networks. Michael A. Arbib (Ed.), MIT Press, 3361.","journal-title":"The Handbook of Brain Theory and Neural Networks"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.571"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605573.3605613"},{"key":"e_1_3_2_58_2","volume-title":"Multilevel-in-Time Methods for Optimal Control of PDEs and Training of Recurrent Neural Networks","author":"Lin Shengchao","year":"2022","unstructured":"Shengchao Lin. 2022. Multilevel-in-Time Methods for Optimal Control of PDEs and Training of Recurrent Neural Networks. Ph.D. Dissertation. Rice University."},{"key":"e_1_3_2_59_2","unstructured":"Yiping Lu Zhuohan Li Di He Zhiqing Sun Bin Dong Tao Qin Liwei Wang and Tie-Yan Liu. 2019. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv:1906.02762. Retrieved from https:\/\/arxiv.org\/abs\/1906.02762"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1137\/17M1144350"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1873951.1874254"},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","unstructured":"Katarzyna Micha\u0142owska Somdatta Goswami George Em Karniadakis and Signe Riemer-S\u00f8rensen. 2023. Neural operator learning for long-time integration in dynamical systems with recurrent neural networks. arXiv:2303.02243. Retrieved from https:\/\/arxiv.org\/abs\/2303.02243","DOI":"10.1109\/IJCNN60899.2024.10650331"},{"key":"e_1_3_2_63_2","volume-title":"International Conference on Learning Representations","author":"Moon Euhyun","year":"2022","unstructured":"Euhyun Moon and Eric C. Cyr. 2022. Parallel training of GRU networks with a multi-grid solver for long sequences. In International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=N1WI0vJLER"},{"key":"e_1_3_2_64_2","first-page":"1","volume-title":"27th ACM Symposium on Operating Systems Principles","author":"Narayanan Deepak","year":"2019","unstructured":"Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In 27th ACM Symposium on Operating Systems Principles, 1\u201315."},{"issue":"1","key":"e_1_3_2_65_2","first-page":"1","article-title":"Applications of time parallelization","volume":"23","author":"Ong Benjamin W.","year":"2020","unstructured":"Benjamin W. Ong and Jacob B. Schroder. 2020. Applications of time parallelization. Computer Vision Science 23, 1 (2020), 1\u201315.","journal-title":"Computer Vision Science"},{"key":"e_1_3_2_66_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_67_2","unstructured":"OpenMPI-FAQ. 2024. FAQ: Running Cuda-Aware Open MPI. Retrieved December 13 2024 from https:\/\/www.open-mpi.org\/faq\/?category=runcuda#mpi-cuda-aware-support"},{"key":"e_1_3_2_68_2","first-page":"1310","volume-title":"International Conference on Machine Learning. PMLR","author":"Pascanu Razvan","year":"2013","unstructured":"Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning. PMLR, 1310\u20131318."},{"key":"e_1_3_2_69_2","unstructured":"Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. Technical report. Retrieved from https:\/\/openreview.net\/forum?id=BJJsrmfCZ"},{"key":"e_1_3_2_70_2","unstructured":"A. Queiruga N. Benjamin Erichson Liam Hodgkinson and Michael W. Mahoney. 2021. Compressing deep ODE-nets using basis function expansions. arXiv:2106.10820. Retrieved from https:\/\/arxiv.org\/abs\/2106.10820"},{"issue":"8","key":"e_1_3_2_71_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.24432\/C54S4K"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611971057.ch4"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1137\/18M1226208"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-0427(00)00516-1"},{"key":"e_1_3_2_76_2","volume-title":"AAAI Conference on Artificial Intelligence","volume":"31","author":"Szegedy Christian","year":"2017","unstructured":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. 2017. Inception-v4, inception-ResNet and the impact of residual connections on learning. In AAAI Conference on Artificial Intelligence, Vol. 31."},{"key":"e_1_3_2_77_2","unstructured":"TorchBraid. [n.d.]. XBraid Interface to PyTorch. Retrieved from https:\/\/github.com\/Multilevel-NN\/torchbraid"},{"key":"e_1_3_2_78_2","unstructured":"Tox. [n.d.]. Tox\u2014Automation Project. Retrieved from https:\/\/tox.wiki\/"},{"key":"e_1_3_2_79_2","volume-title":"Multigrid","author":"Trottenberg U.","year":"2001","unstructured":"U. Trottenberg, C. Oosterlee, and A. Sch \\(\\ddot{\\mbox{u}}\\) Ller. 2001. Multigrid. Academic Press, London, UK."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1137\/080727890"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298801"},{"key":"e_1_3_2_82_2","unstructured":"XBraid. [n.d.]. XBraid: Parallel Multigrid in Time. Retrieved from https:\/\/github.com\/XBraid\/xbraid\/"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0962492917000083"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco_a_01199"},{"key":"e_1_3_2_85_2","unstructured":"Qian-Yi Zhou Jaesik Park and Vladlen Koltun. 2018. Open3D: A modern library for 3D data processing. arXiv:1801.09847. Retrieved from https:\/\/arxiv.org\/abs\/1801.09847"}],"container-title":["ACM Transactions on Mathematical Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3759244","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3759244","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T16:29:24Z","timestamp":1759163364000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3759244"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,29]]},"references-count":84,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3759244"],"URL":"https:\/\/doi.org\/10.1145\/3759244","relation":{},"ISSN":["0098-3500","1557-7295"],"issn-type":[{"type":"print","value":"0098-3500"},{"type":"electronic","value":"1557-7295"}],"subject":[],"published":{"date-parts":[[2025,9,29]]},"assertion":[{"value":"2023-12-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}