{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T08:36:41Z","timestamp":1759826201738,"version":"3.37.3"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T00:00:00Z","timestamp":1698364800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T00:00:00Z","timestamp":1698364800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE-AC02-07CH11359"],"award-info":[{"award-number":["DE-AC02-07CH11359"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["1934700"],"award-info":[{"award-number":["1934700"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Softw Big Sci"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We study the performance of a cloud-based GPU-accelerated inference server to speed up event reconstruction in neutrino data batch jobs. Using detector data from the ProtoDUNE experiment and employing the standard DUNE grid job submission tools, we attempt to reprocess the data by running several thousand concurrent grid jobs, a rate we expect to be typical of current and future neutrino physics experiments. We process most of the dataset with the GPU version of our processing algorithm and the remainder with the CPU version for timing comparisons. We find that a 100-GPU cloud-based server is able to easily meet the processing demand, and that using the GPU version of the event processing algorithm is two times faster than processing these data with the CPU version when comparing to the newest CPUs in our sample. The amount of data transferred to the inference server during the GPU runs can overwhelm even the highest-bandwidth network switches, however, unless care is taken to observe network facility limits or otherwise distribute the jobs to multiple sites. We discuss the lessons learned from this processing campaign and several avenues for future improvements.<\/jats:p>","DOI":"10.1007\/s41781-023-00101-0","type":"journal-article","created":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T13:02:09Z","timestamp":1698411729000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Accelerating Machine Learning Inference with GPUs in ProtoDUNE Data Processing"],"prefix":"10.1007","volume":"7","author":[{"given":"Tejin","family":"Cai","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kenneth","family":"Herner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tingjun","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Acosta Flechas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philip","family":"Harris","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Burt","family":"Holzman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin","family":"Pedro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nhan","family":"Tran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,10,27]]},"reference":[{"issue":"7","key":"101_CR1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/898\/7\/072053","volume":"898","author":"F Psihas","year":"2017","unstructured":"Psihas F (2017) The convolutional visual network for identification and reconstruction of NOvA events. J Phys Conf Ser 898(7):072053. https:\/\/doi.org\/10.1088\/1742-6596\/898\/7\/072053","journal-title":"J Phys Conf Ser"},{"issue":"11","key":"101_CR2","doi-asserted-by":"publisher","first-page":"11020","DOI":"10.1088\/1748-0221\/13\/11\/P11020","volume":"13","author":"GN Perdue","year":"2018","unstructured":"Perdue GN et al (2018) Reducing model bias in a deep learning classifier using domain adversarial neural networks in the MINERvA experiment. JINST 13(11):11020. https:\/\/doi.org\/10.1088\/1748-0221\/13\/11\/P11020. arXiv:1808.08332 [physics.data-an]","journal-title":"JINST"},{"key":"101_CR3","doi-asserted-by":"publisher","unstructured":"Racah E, Ko S, Sadowski P, Bhimji W, Tull C, Oh S-Y, Baldi P, Prabhat (2016). Revealing fundamental physics from the Daya Bay neutrino experiment using deep neural networks. 15th IEEE International Conference on Machine Learning and Applications (ICMLA). https:\/\/doi.org\/10.1109\/ICMLA.2016.0160. arXiv:1601.07621 [stat.ML]","DOI":"10.1109\/ICMLA.2016.0160"},{"key":"101_CR4","doi-asserted-by":"publisher","first-page":"07041","DOI":"10.1088\/1748-0221\/16\/07\/P07041","volume":"16","author":"R Abbasi","year":"2021","unstructured":"Abbasi R et al (2021) A convolutional neural network based cascade reconstruction for the IceCube neutrino observatory. JINST 16:07041. https:\/\/doi.org\/10.1088\/1748-0221\/16\/07\/P07041. arXiv:2101.11589 [hep-ex]","journal-title":"JINST"},{"issue":"08","key":"101_CR5","doi-asserted-by":"publisher","first-page":"08008","DOI":"10.1088\/1748-0221\/15\/08\/T08008","volume":"15","author":"B Abi","year":"2020","unstructured":"Abi B et al (2020) Deep Underground Neutrino Experiment (DUNE), Far Detector technical design report, volume I. Introduction to DUNE. JINST 15(08):08008. https:\/\/doi.org\/10.1088\/1748-0221\/15\/08\/T08008. arXiv:2002.02967 [physics.ins-det]","journal-title":"JINST"},{"key":"101_CR6","unstructured":"Antonello M et al (2015) A proposal for a three detector short-baseline neutrino oscillation program in the Fermilab booster neutrino beam. arXiv:1503.01520 [physics.ins-det]"},{"issue":"11","key":"101_CR7","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.105.112003","volume":"105","author":"P Abratenko","year":"2022","unstructured":"Abratenko P et al (2022) Search for an anomalous excess of charged-current quasielastic $$\\nu$$e interactions with the MicroBooNE experiment using Deep-Learning-based reconstruction. Phys Rev D 105(11):112003. https:\/\/doi.org\/10.1103\/PhysRevD.105.112003. arXiv:2110.14080 [hep-ex]","journal-title":"Phys Rev D"},{"issue":"01","key":"101_CR8","doi-asserted-by":"publisher","first-page":"01018","DOI":"10.1088\/1748-0221\/17\/01\/P01018","volume":"17","author":"R Acciarri","year":"2022","unstructured":"Acciarri R et al (2022) A deep-learning based raw waveform region-of-interest finder for the liquid argon time projection chamber. JINST 17(01):01018. https:\/\/doi.org\/10.1088\/1748-0221\/17\/01\/P01018. arXiv:2103.06391 [physics.ins-det]","journal-title":"JINST"},{"issue":"10","key":"101_CR9","doi-asserted-by":"publisher","first-page":"903","DOI":"10.1140\/epjc\/s10052-022-10791-2","volume":"82","author":"A Abed Abud","year":"2022","unstructured":"Abed Abud A et al (2022) Separation of track- and shower-like energy deposits in ProtoDUNE-SP using a convolutional neural network. Eur Phys J C 82(10):903. https:\/\/doi.org\/10.1140\/epjc\/s10052-022-10791-2. arXiv:2203.17053 [physics.ins-det]","journal-title":"Eur Phys J C"},{"key":"101_CR10","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2020.604083","volume":"3","author":"M Wang","year":"2021","unstructured":"Wang M, Yang T, Acosta Flechas M, Harris P, Hawks B, Holzman B, Knoepfel K, Krupa J, Pedro K, Tran N (2021) GPU-accelerated machine learning inference as a service for computing in neutrino experiments. Front Big Data 3:604083. https:\/\/doi.org\/10.3389\/fdata.2020.604083. arXiv:2009.04509 [physics.comp-ph]","journal-title":"Front Big Data"},{"issue":"12","key":"101_CR11","doi-asserted-by":"publisher","first-page":"12004","DOI":"10.1088\/1748-0221\/15\/12\/P12004","volume":"15","author":"B Abi","year":"2020","unstructured":"Abi B et al (2020) First results on ProtoDUNE-SP liquid argon time projection chamber performance from a beam test at the CERN Neutrino Platform. JINST 15(12):12004. https:\/\/doi.org\/10.1088\/1748-0221\/15\/12\/P12004. arXiv:2007.06722 [physics.ins-det]","journal-title":"JINST"},{"issue":"01","key":"101_CR12","doi-asserted-by":"publisher","first-page":"01005","DOI":"10.1088\/1748-0221\/17\/01\/P01005","volume":"17","author":"AA Abud","year":"2022","unstructured":"Abud AA et al (2022) Design, construction and operation of the ProtoDUNE-SP Liquid Argon TPC. JINST 17(01):01005. https:\/\/doi.org\/10.1088\/1748-0221\/17\/01\/P01005. arXiv:2108.01902 [physics.ins-det]","journal-title":"JINST"},{"issue":"1","key":"101_CR13","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/888\/1\/012038","volume":"888","author":"F Pietropaolo","year":"2017","unstructured":"Pietropaolo F (2017) Review of liquid-argon detectors development at the CERN neutrino platform. J Phys Conf Ser 888(1):012038. https:\/\/doi.org\/10.1088\/1742-6596\/888\/1\/012038","journal-title":"J Phys Conf Ser"},{"issue":"9","key":"101_CR14","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1140\/epjc\/s10052-015-3659-3","volume":"75","author":"JS Marshall","year":"2015","unstructured":"Marshall JS, Thomson MA (2015) The Pandora software development kit for pattern recognition. Eur Phys J C 75(9):439. https:\/\/doi.org\/10.1140\/epjc\/s10052-015-3659-3. arXiv:1506.05348 [physics.data-an]","journal-title":"Eur Phys J C"},{"issue":"1","key":"101_CR15","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1140\/epjc\/s10052-017-5481-6","volume":"78","author":"R Acciarri","year":"2018","unstructured":"Acciarri R et al (2018) The Pandora multi-algorithm approach to automated pattern recognition of cosmic-ray muon and neutrino events in the MicroBooNE detector. Eur Phys J C 78(1):82. https:\/\/doi.org\/10.1140\/epjc\/s10052-017-5481-6. arXiv:1708.03135 [hep-ex]","journal-title":"Eur Phys J C"},{"key":"101_CR16","doi-asserted-by":"publisher","unstructured":"Abed Abud A et al (2023) Reconstruction of interactions in the ProtoDUNE-SP detector with Pandora. Eur Phys J C 83(7):618. https:\/\/doi.org\/10.1140\/epjc\/s10052-023-11733-2. arXiv:2206.14521 [hep-ex]","DOI":"10.1140\/epjc\/s10052-023-11733-2"},{"key":"101_CR17","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. https:\/\/www.tensorflow.org\/"},{"key":"101_CR18","unstructured":"The LArSoft Collaboration: The larrecodnn module. https:\/\/github.com\/LArSoft\/larrecodnn. Accessed: 17 Oct 2022 (2022)"},{"key":"101_CR19","unstructured":"NVIDIA: NVIDIA Triton Inference Server. https:\/\/developer.nvidia.com\/nvidia-triton-inference-server. Accessed: 05 May 2022 (2022)"},{"key":"101_CR20","unstructured":"Google: Google Kubernetes Engine. https:\/\/cloud.google.com\/kubernetes-engine. Accessed: 19 Jul 2022 (2022)"},{"key":"101_CR21","unstructured":"Google: GKE Ingress for HTTP(S) Load Balancing. https:\/\/cloud.google.com\/kubernetes-engine\/docs\/concepts\/ingress. Accessed: 19 Jul 2022 (2022)"},{"key":"101_CR22","unstructured":"NVIDIA: NVIDIA Triton Inference Server-Metrics summary. https:\/\/github.com\/triton-inference-server\/server\/blob\/main\/docs\/user_guide\/metrics.md. Accessed: 2023-03-30 (2023)"},{"key":"101_CR23","unstructured":"Prometheus Authors: Exposition Formats. https:\/\/prometheus.io\/docs\/instrumenting\/exposition_formats\/. Accessed: 30 Mar 2023 (2023)"},{"issue":"4","key":"101_CR24","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/898\/4\/042057","volume":"898","author":"EL Snider","year":"2017","unstructured":"Snider EL, Petrillo G (2017) LArSoft: toolkit for simulation, reconstruction and analysis of liquid argon TPC neutrino detectors. J Phys Conf Ser 898(4):042057. https:\/\/doi.org\/10.1088\/1742-6596\/898\/4\/042057","journal-title":"J Phys Conf Ser"},{"key":"101_CR25","doi-asserted-by":"publisher","first-page":"03019","DOI":"10.1051\/epjconf\/202024503019","volume":"245","author":"K Herner","year":"2020","unstructured":"Herner K (2020) DUNE production processing and workflow management software evaluation. Eur Phys J Web Conf 245:03019. https:\/\/doi.org\/10.1051\/epjconf\/202024503019","journal-title":"Eur Phys J Web Conf"},{"key":"101_CR26","doi-asserted-by":"publisher","first-page":"03024","DOI":"10.1051\/epjconf\/202024503024","volume":"245","author":"M Mengel","year":"2020","unstructured":"Mengel M, White S, Podstavkov V, Wiersma M, Mazzacane A, Herner K (2020) Production Operations Management System (POMS) for Fermilab Experiments. Eur Phys J Web Conf 245:03024. https:\/\/doi.org\/10.1051\/epjconf\/202024503024","journal-title":"Eur Phys J Web Conf"},{"key":"101_CR27","doi-asserted-by":"publisher","unstructured":"Pordes R, OSG Consortium Petravick D, Kramer B, OlsonD, Livny M, Roy A, Avery P, Blackburn K, Wenaus T, W\u00fcrthwein F, Foster I, Gardner R, Wilde M, Blatecky A, McGee J, Quick R (2007) The Open Science Grid. J Phys Conf Ser 78:012057. https:\/\/doi.org\/10.1088\/1742-6596\/78\/1\/012057","DOI":"10.1088\/1742-6596\/78\/1\/012057"},{"key":"101_CR28","volume-title":"Grid computing: making the global infrastructure a reality","author":"D Thain","year":"2002","unstructured":"Thain D, Tannenbaum T, Livny M (2002) Condor and the grid. In: Berman F, Fox G, Hey T (eds) Grid computing: making the global infrastructure a reality. Wiley, Hoboken, NJ"},{"key":"101_CR29","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2020.101213","author":"B Bockelman","year":"2020","unstructured":"Bockelman B, Livny M, Lin B, Prelz F (2020) Principles, technologies, and time: the translational journey of the HTCondor-CE. J Comput Sci. https:\/\/doi.org\/10.1016\/j.jocs.2020.101213","journal-title":"J Comput Sci"},{"issue":"1","key":"101_CR30","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/s41781-019-0026-3","volume":"3","author":"M Barisits","year":"2019","unstructured":"Barisits M, Beermann T, Berghaus F, Bockelman B, Bogado J, Cameron D, Christidis D, Ciangottini D, Dimitrov G, Elsing M, Garonne V, di Girolamo A, Goossens L, Guan W, Guenther J, Javurek T, Kuhn D, Lassnig M, Lopez F, Magini N, Molfetas A, Nairz A, Ould-Saada F, Prenner S, Serfon C, Stewart G, Vaandering E, Vasileva P, Vigne R, Wegner T (2019) Rucio: scientific data management. Comput Softw Big Sci 3(1):11. https:\/\/doi.org\/10.1007\/s41781-019-0026-3","journal-title":"Comput Softw Big Sci"}],"container-title":["Computing and Software for Big Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-023-00101-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41781-023-00101-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-023-00101-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T13:06:12Z","timestamp":1698411972000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41781-023-00101-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,27]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["101"],"URL":"https:\/\/doi.org\/10.1007\/s41781-023-00101-0","relation":{},"ISSN":["2510-2036","2510-2044"],"issn-type":[{"type":"print","value":"2510-2036"},{"type":"electronic","value":"2510-2044"}],"subject":[],"published":{"date-parts":[[2023,10,27]]},"assertion":[{"value":"1 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 July 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interests"}}],"article-number":"11"}}