{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T11:28:43Z","timestamp":1783164523400,"version":"3.54.6"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,7,18]],"date-time":"2024-07-18T00:00:00Z","timestamp":1721260800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,18]],"date-time":"2024-07-18T00:00:00Z","timestamp":1721260800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE-SC0010005"],"award-info":[{"award-number":["DE-SC0010005"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006230","name":"Fermilab","doi-asserted-by":"publisher","award":["DE-AC02-07CH11359"],"award-info":[{"award-number":["DE-AC02-07CH11359"]}],"id":[{"id":"10.13039\/100006230","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["ACI-1450377"],"award-info":[{"award-number":["ACI-1450377"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Softw Big Sci"],"published-print":{"date-parts":[[2024,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With machine learning applications now spanning a variety of computational tasks, multi-user shared computing facilities are devoting a rapidly increasing proportion of their resources to such algorithms. Graph neural networks (GNNs), for example, have provided astounding improvements in extracting complex signatures from data and are now widely used in a variety of applications, such as particle jet classification in high energy physics (HEP). However, GNNs also come with an enormous computational penalty that requires the use of GPUs to maintain reasonable throughput. At shared computing facilities, such as those used by physicists at Fermi National Accelerator Laboratory (Fermilab), methodical resource allocation and high throughput at the many-user scale are key to ensuring that resources are being used as efficiently as possible. These facilities, however, primarily provide CPU-only nodes, which proves detrimental to time-to-insight and computational throughput for workflows that include machine learning inference. In this work, we describe how a shared computing facility can use the NVIDIA Triton Inference Server to optimize its resource allocation and computing structure, recovering high throughput while scaling out to multiple users by massively parallelizing their machine learning inference. To demonstrate the effectiveness of this system in a realistic multi-user environment, we use the Fermilab Elastic Analysis Facility augmented with the Triton Inference Server to provide scalable and high-throughput access to a HEP-specific GNN and report on the outcome.<\/jats:p>","DOI":"10.1007\/s41781-024-00123-2","type":"journal-article","created":{"date-parts":[[2024,7,18]],"date-time":"2024-07-18T04:01:36Z","timestamp":1721275296000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Optimizing High-Throughput Inference on Graph Neural Networks at Shared Computing Facilities with the NVIDIA Triton Inference Server"],"prefix":"10.1007","volume":"8","author":[{"given":"Claire","family":"Savard","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Manganelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Burt","family":"Holzman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lindsey","family":"Gray","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexx","family":"Perloff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Pedro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Stenson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keith","family":"Ulmer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,18]]},"reference":[{"issue":"2","key":"123_CR1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1085\/2\/022008","volume":"1085","author":"K Albertsson","year":"2018","unstructured":"Albertsson K, Altoe P, Anderson D et al (2018) Machine learning in high energy physics community white paper. J Phys Conf Ser 1085(2):022008. https:\/\/doi.org\/10.1088\/1742-6596\/1085\/2\/022008","journal-title":"J Phys Conf Ser"},{"issue":"1","key":"123_CR2","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1146\/annurev-nucl-101917-021019","volume":"68","author":"D Guest","year":"2018","unstructured":"Guest D, Cranmer K, Whiteson D (2018) Deep learning and its application to LHC physics. Ann Rev Nucl Part Sci 68(1):161\u2013181. https:\/\/doi.org\/10.1146\/annurev-nucl-101917-021019","journal-title":"Ann Rev Nucl Part Sci"},{"key":"123_CR3","doi-asserted-by":"publisher","DOI":"10.1109\/CEIT.2018.8751930","author":"E Buber","year":"2018","unstructured":"Buber E, Diri B (2018) Performance analysis and CPU vs GPU comparison for deep learning. Int Conf Control Eng Inf Technol Proc. https:\/\/doi.org\/10.1109\/CEIT.2018.8751930","journal-title":"Int Conf Control Eng Inf Technol Proc"},{"key":"123_CR4","unstructured":"Wang Y, Wei G-Y, Brooks D (2019) Benchmarking TPU, GPU, and CPU platforms for deep learning. arXiv:1907.10701 [cs.LG]"},{"key":"123_CR5","unstructured":"Baker M, Fox GC, Yau HW (1995) Cluster computing review. Northeast Parallel Architecture Center. 33"},{"key":"123_CR6","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1016\/j.nuclphysbps.2007.11.121","volume":"177\u2013178","author":"I Bloch","year":"2008","unstructured":"Bloch I (2008) The LHC physics center. Nucl Phys B Proc Suppl 177\u2013178:261\u2013262. https:\/\/doi.org\/10.1016\/j.nuclphysbps.2007.11.121","journal-title":"Nucl Phys B Proc Suppl"},{"key":"123_CR7","doi-asserted-by":"publisher","first-page":"08003","DOI":"10.1088\/1748-0221\/3\/08\/S08003","volume":"3","author":"ATLAS Collaboration","year":"2008","unstructured":"ATLAS Collaboration (2008) The ATLAS experiment at the CERN large hadron collider. JINST 3:08003. https:\/\/doi.org\/10.1088\/1748-0221\/3\/08\/S08003","journal-title":"JINST"},{"key":"123_CR8","doi-asserted-by":"publisher","first-page":"08004","DOI":"10.1088\/1748-0221\/3\/08\/S08004","volume":"3","author":"CMS Collaboration","year":"2008","unstructured":"CMS Collaboration (2008) The CMS experiment at the CERN LHC. JINST 3:08004. https:\/\/doi.org\/10.1088\/1748-0221\/3\/08\/S08004","journal-title":"JINST"},{"key":"123_CR9","doi-asserted-by":"publisher","unstructured":"Marinescu DC (2023) Chapter 3-parallel processing and distributed computing. In: Cloud computing, 3rd edn. Morgan Kaufmann, Burlington, p 41\u201394. https:\/\/doi.org\/10.1016\/B978-0-32-385277-7.00010-5","DOI":"10.1016\/B978-0-32-385277-7.00010-5"},{"key":"123_CR10","unstructured":"NVIDIA Corporation. Triton inference server: an optimized cloud and edge inferencing solution. https:\/\/github.com\/triton-inference-server. Accessed 8 Dec 2023"},{"key":"123_CR11","unstructured":"Paszke A, Gross S, Massa F et al (2019) PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol 32. Curran Associates Inc., Red Hook, p 8024\u20138035. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf. Accessed 8 Dec 2023"},{"key":"123_CR12","unstructured":"Abadi M, Agarwal A, Barham P et al (2015) TensorFlow: large-scale machine learning on heterogeneous systems. https:\/\/www.tensorflow.org\/. Accessed 8 Dec 2023"},{"key":"123_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.peva.2020.102183","volume":"147","author":"Y Inoue","year":"2021","unstructured":"Inoue Y (2021) Queueing analysis of GPU-based inference servers with dynamic batching: a closed-form characterization. Perform Eval 147:102183. https:\/\/doi.org\/10.1016\/j.peva.2020.102183","journal-title":"Perform Eval"},{"key":"123_CR14","unstructured":"NVIDIA Corporation. Triton architecture. https:\/\/docs.nvidia.com\/deeplearning\/triton-inference-server\/user-guide\/docs\/user_guide\/architecture.html. Accessed 8 Dec 2023"},{"key":"123_CR15","unstructured":"OKD. The community distribution of Kubernetes that powers Red Hat OpenShift. https:\/\/www.okd.io. Accessed 8 Dec 2023"},{"key":"123_CR16","unstructured":"Red Hat OpenShift. https:\/\/www.redhat.com\/en\/technologies\/cloud-computing\/openshift. Accessed 8 Dec 2023"},{"key":"123_CR17","unstructured":"kubernetes. https:\/\/kubernetes.io. Accessed 8 Dec 2023"},{"issue":"2\u20134","key":"123_CR18","first-page":"323","volume":"17","author":"HTCondor Team","year":"2005","unstructured":"HTCondor Team (2005) Distributed computing in practice: the Condor experience. Concurr Comput Pract Exp 17(2\u20134):323\u2013356","journal-title":"Concurr Comput Pract Exp"},{"key":"123_CR19","unstructured":"CRAB Server. https:\/\/github.com\/dmwm\/CRABServer. Accessed 8 Dec 2023"},{"key":"123_CR20","unstructured":"Dask. https:\/\/github.com\/dask\/dask. Accessed 8 Dec 2023"},{"key":"123_CR21","unstructured":"MinIO. High performance object storage for AI. https:\/\/min.io. Accessed 8 Dec 2023"},{"key":"123_CR22","unstructured":"gRPC, a high performance, open source universal RPC framework. https:\/\/grpc.io. Accessed 8 Dec 2023"},{"key":"123_CR23","unstructured":"HAProxy. The reliable high performance TCP\/HTTP load balancer. https:\/\/www.haproxy.org. Accessed 8 Dec 2023"},{"key":"123_CR24","unstructured":"nginx. https:\/\/nginx.org\/en. Accessed 8 Dec 2023"},{"key":"123_CR25","unstructured":"Prometheus monitoring system & time series database. https:\/\/prometheus.io. Accessed 8 Dec 2023"},{"key":"123_CR26","unstructured":"Grafana Labs. https:\/\/grafana.com. Accessed 8 Dec 2023"},{"issue":"5","key":"123_CR27","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.101.056019","volume":"101","author":"H Qu","year":"2020","unstructured":"Qu H, Gouskos L (2020) ParticleNet: jet tagging via particle clouds. Phys Rev D 101(5):056019. https:\/\/doi.org\/10.1103\/PhysRevD.101.056019. arXiv:1902.08570 [hep-ph]","journal-title":"Phys Rev D"},{"key":"123_CR28","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p 770\u2013778. https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"123_CR29","unstructured":"Canziani A, Paszke A, Culurciello E (2017) An analysis of deep neural network models for practical applications. arXiv:1605.07678 [cs.CV]"},{"key":"123_CR30","doi-asserted-by":"publisher","unstructured":"Chen T, Guestrin C (2016) XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD \u201916. ACM, San Francisco. https:\/\/doi.org\/10.1145\/2939672.2939785","DOI":"10.1145\/2939672.2939785"},{"key":"123_CR31","doi-asserted-by":"publisher","unstructured":"Rankin DS et al (2020) FPGAs-as-a-service toolkit (FaaST). In: 2020 IEEE\/ACM International Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC), p 38\u201347. https:\/\/doi.org\/10.1109\/H2RC51942.2020.00010","DOI":"10.1109\/H2RC51942.2020.00010"},{"key":"123_CR32","unstructured":"Kubernetes. Horizontal pod autoscaling. https:\/\/kubernetes.io\/docs\/tasks\/run-application\/horizontal-pod-autoscale\/. Accessed 8 Dec 2023"}],"container-title":["Computing and Software for Big Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-024-00123-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41781-024-00123-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-024-00123-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,18]],"date-time":"2024-07-18T04:20:21Z","timestamp":1721276421000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41781-024-00123-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,18]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["123"],"URL":"https:\/\/doi.org\/10.1007\/s41781-024-00123-2","relation":{},"ISSN":["2510-2036","2510-2044"],"issn-type":[{"value":"2510-2036","type":"print"},{"value":"2510-2044","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,18]]},"assertion":[{"value":"8 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"14"}}