{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T08:39:54Z","timestamp":1759826394294,"version":"3.37.3"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,14]],"date-time":"2021-12-14T00:00:00Z","timestamp":1639440000000},"content-version":"vor","delay-in-days":13,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012470","name":"CERN","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012470","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Softw Big Sci"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The HEPiX Benchmarking Working Group has developed a framework to benchmark the performance of a computational server using the software applications of the High Energy Physics (HEP) community. This framework consists of two main components, named HEP-Workloads and HEPscore. HEP-Workloads is a collection of standalone production applications provided by a number of HEP experiments. HEPscore is designed to run HEP-Workloads and provide an overall measurement that is representative of the computing power of a system. HEPscore is able to measure the performance of systems with different processor architectures and accelerators. The framework is completed by the HEP Benchmark Suite that simplifies the process of executing HEPscore and other benchmarks such as HEP-SPEC06, SPEC CPU 2017, and DB12. This paper describes the motivation, the design choices, and the results achieved by the HEPiX Benchmarking Working group. A perspective on future plans is also presented.<\/jats:p>","DOI":"10.1007\/s41781-021-00074-y","type":"journal-article","created":{"date-parts":[[2021,12,14]],"date-time":"2021-12-14T12:16:54Z","timestamp":1639484214000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["HEPiX Benchmarking Solution for WLCG Computing Resources"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9789-3188","authenticated-orcid":false,"given":"Domenico","family":"Giordano","sequence":"first","affiliation":[]},{"given":"Manfred","family":"Alef","sequence":"additional","affiliation":[]},{"given":"Luca","family":"Atzori","sequence":"additional","affiliation":[]},{"given":"Jean-Michel","family":"Barbet","sequence":"additional","affiliation":[]},{"given":"Olga","family":"Datskova","sequence":"additional","affiliation":[]},{"given":"Maria","family":"Girone","sequence":"additional","affiliation":[]},{"given":"Christopher","family":"Hollowell","sequence":"additional","affiliation":[]},{"given":"Martina","family":"Javurkova","sequence":"additional","affiliation":[]},{"given":"Riccardo","family":"Maganza","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0436-2183","authenticated-orcid":false,"given":"Miguel F.","family":"Medeiros","sequence":"additional","affiliation":[]},{"given":"Michele","family":"Michelotto","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9608-9940","authenticated-orcid":false,"given":"Lorenzo","family":"Rinaldi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4371-1430","authenticated-orcid":false,"given":"Andrea","family":"Sciab\u00e0","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7430-7599","authenticated-orcid":false,"given":"Randall J.","family":"Sobie","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4229-7191","authenticated-orcid":false,"given":"David","family":"Southwick","sequence":"additional","affiliation":[]},{"given":"Tristan","family":"Sullivan","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9322-9565","authenticated-orcid":false,"given":"Andrea","family":"Valassi","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,12,14]]},"reference":[{"key":"74_CR1","doi-asserted-by":"publisher","first-page":"052009","DOI":"10.1088\/1742-6596\/219\/5\/052009","volume":"219","author":"M Michelotto","year":"2010","unstructured":"Michelotto M et al (2010) A comparison of HEP code with SPEC1 benchmarks on multi-core worker nodes. J Phys Conf Ser 219:052009. https:\/\/doi.org\/10.1088\/1742-6596\/219\/5\/052009","journal-title":"J Phys Conf Ser"},{"key":"74_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1186736.1186737","volume":"34","author":"JL Henning","year":"2016","unstructured":"Henning JL (2016) SPEC CPU2006 benchmark descriptions. SIGARCH Comput Archit News 34:1. https:\/\/doi.org\/10.1145\/1186736.1186737","journal-title":"SIGARCH Comput Archit News"},{"key":"74_CR3","unstructured":"WLCG (2021) The Worldwide LHC Computing Grid. http:\/\/wlcg.web.cern.ch. Accessed 11 Nov 2021"},{"key":"74_CR4","doi-asserted-by":"publisher","first-page":"082011","DOI":"10.1088\/1742-6596\/898\/8\/082011","volume":"898","author":"P Charpentier","year":"2017","unstructured":"Charpentier P (2017) Benchmarking worker nodes using LHCb productions and comparing with HEP-SPEC06. J Phys Conf Ser 898:082011. https:\/\/doi.org\/10.1088\/1742-6596\/898\/8\/082011","journal-title":"J Phys Conf Ser"},{"key":"74_CR5","unstructured":"HEPiX Benchmarking Working Group (2021). https:\/\/w3.hepix.org\/benchmarking.html. Accessed 11 Nov 2021"},{"key":"74_CR6","unstructured":"SPEC CPU (2017) Standard Performance Evaluation Corporation. https:\/\/www.spec.org\/cpu2017\/. Accessed 11 Nov 2021"},{"key":"74_CR7","doi-asserted-by":"publisher","first-page":"08011","DOI":"10.1051\/epjconf\/201921408011","volume":"214","author":"D Giordano","year":"2019","unstructured":"Giordano D, Alef M, Michelotto M (2019) Next generation of HEP CPU benchmarks. EPJ Web Conf 214:08011. https:\/\/doi.org\/10.1051\/epjconf\/201921408011","journal-title":"EPJ Web Conf"},{"key":"74_CR8","unstructured":"Giordano D (2021) HS06 -m32 Vs -m64. HEPiX Benchmarking Working Group. https:\/\/indico.cern.ch\/event\/624834\/contributions\/2646335\/attachments\/1486043\/2307590\/HS06_32_Vs_64_bits-giordano.pdf. Accessed 11 Nov 2021"},{"key":"74_CR9","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/664\/2\/022034","author":"G Roy","year":"2015","unstructured":"Roy G et al (2015) Evaluation of containers as a virtualisation alternative for HEP workloads. J Phys Conf Ser. https:\/\/doi.org\/10.1088\/1742-6596\/664\/2\/022034","journal-title":"J Phys Conf Ser"},{"key":"74_CR10","doi-asserted-by":"publisher","first-page":"08024","DOI":"10.1051\/epjconf\/201921408024","volume":"214","author":"S Muralidharan","year":"2019","unstructured":"Muralidharan S, Smith D (2019) Trident: an automated system tool for collecting and analyzing performance counters. EPJ Web Conf 214:08024. https:\/\/doi.org\/10.1051\/epjconf\/201921408024","journal-title":"EPJ Web Conf"},{"key":"74_CR11","doi-asserted-by":"publisher","first-page":"012073","DOI":"10.1088\/1742-6596\/1525\/1\/012073","volume":"1525","author":"D Giordano","year":"2020","unstructured":"Giordano D, Santorinaiou E (2020) Next generation of HEP CPU benchmarks. J Phys Conf Ser 1525:012073. https:\/\/doi.org\/10.1088\/1742-6596\/1525\/1\/012073","journal-title":"J Phys Conf Ser"},{"key":"74_CR12","unstructured":"Giordano D (2017) Benchmark Working Group Update. WLCG Workshop in Manchester. https:\/\/indico.cern.ch\/event\/609911\/contributions\/2620190\/attachments\/1480455\/2295576\/WLCG_Workshop_2017_benchmarking_giordano.pdf. Accessed 11 Nov 2021"},{"key":"74_CR13","doi-asserted-by":"publisher","first-page":"07035","DOI":"10.1051\/epjconf\/202024507035","volume":"245","author":"A Valassi","year":"2020","unstructured":"Valassi A et al (2020) Using HEP experiment workflows for the benchmarking and accounting of WLCG computing resources. EPJ Web Conf 245:07035. https:\/\/doi.org\/10.1051\/epjconf\/202024507035","journal-title":"EPJ Web Conf"},{"key":"74_CR14","unstructured":"HEP-Benchmarks Repository (2021). https:\/\/gitlab.cern.ch\/hep-benchmarks Accessed 11 Nov 2021"},{"key":"74_CR15","doi-asserted-by":"publisher","first-page":"062031","DOI":"10.1088\/1742-6596\/898\/6\/062031","volume":"898","author":"J Blomer","year":"2017","unstructured":"Blomer J et al (2017) New directions in the CernVM file system. J Phys Conf Ser 898:062031. https:\/\/doi.org\/10.1088\/1742-6596\/898\/6\/062031","journal-title":"J Phys Conf Ser"},{"key":"74_CR16","doi-asserted-by":"publisher","unstructured":"A definition of the Monte Carlo applications to generate, simulate, digitize and reconstruct particle physics collision events. In: A roadmap for HEP software and computing R&D for the 2020s computing and software for big science, vol 3, p 1. https:\/\/doi.org\/10.1007\/s41781-018-0018-8","DOI":"10.1007\/s41781-018-0018-8"},{"key":"74_CR17","doi-asserted-by":"publisher","first-page":"S08003","DOI":"10.1088\/1748-0221\/3\/08\/S08003","volume":"3","author":"G Aad","year":"2008","unstructured":"Aad G et al (2008) The ATLAS experiment at the CERN large hadron collider. JINST 3:S08003. https:\/\/doi.org\/10.1088\/1748-0221\/3\/08\/S08003","journal-title":"JINST"},{"key":"74_CR18","doi-asserted-by":"publisher","unstructured":"The CMS Collaboration (2008) The CMS experiment at the CERN LHC. JINST 3:S08004. https:\/\/doi.org\/10.1088\/1748-0221\/3\/08\/S08004","DOI":"10.1088\/1748-0221\/3\/08\/S08004"},{"key":"74_CR19","doi-asserted-by":"publisher","first-page":"S08005","DOI":"10.1088\/1748-0221\/3\/08\/S08005","volume":"3","author":"AA Alves","year":"2008","unstructured":"Alves AA et al (2008) The LHCb detector at the LHC. JINST 3:S08005. https:\/\/doi.org\/10.1088\/1748-0221\/3\/08\/S08005","journal-title":"JINST"},{"key":"74_CR20","doi-asserted-by":"publisher","first-page":"123C01","DOI":"10.1093\/ptep\/ptz106","volume":"12","author":"E Kou","year":"2019","unstructured":"Kou E et al (2019) The Belle II Physics ok. Progr Theor Exp Phys 12:123C01. https:\/\/doi.org\/10.1093\/ptep\/ptz106","journal-title":"Progr Theor Exp Phys"},{"key":"74_CR21","unstructured":"Docker containers (2021). https:\/\/www.docker.com\/resources\/what-container. Accessed: 11 Nov 2021"},{"key":"74_CR22","unstructured":"Singularity (2021). https:\/\/sylabs.io\/singularity\/. Accessed 11 Nov 2021"},{"key":"74_CR23","unstructured":"Atlas Athena license (2021). https:\/\/gitlab.cern.ch\/atlas\/athena\/blob\/master\/LICENSE. Accessed 11 Nov 2021"},{"key":"74_CR24","doi-asserted-by":"publisher","unstructured":"Belle II license (2021). https:\/\/github.com\/belle2\/basf2\/blob\/main\/LICENSE.md. https:\/\/doi.org\/10.5281\/zenodo.5574115. Accessed 11 Nov 2021","DOI":"10.5281\/zenodo.5574115"},{"key":"74_CR25","unstructured":"CMS CMSSW license (2021). https:\/\/github.com\/cms-sw\/cmssw\/blob\/master\/LICENSE. Accessed 11 Nov 2021"},{"key":"74_CR26","unstructured":"LHCb license (2021). https:\/\/gitlab.cern.ch\/lhcb\/Gauss\/-\/blob\/master\/COPYING. Accessed 11 Nov 2021"},{"key":"74_CR27","unstructured":"WLCG HEPscore deployment Task Force (2021). https:\/\/indico.cern.ch\/event\/969947\/. Accessed 11 Nov 2021"},{"key":"74_CR28","doi-asserted-by":"publisher","first-page":"082011","DOI":"10.1088\/1742-6596\/898\/8\/082011","volume":"898","author":"P Charpentier","year":"2017","unstructured":"Charpentier P (2017) Benchmarking worker nodes using LHCb productions and comparing with HEPSpec06. J Phys Conf Ser 898:082011. https:\/\/doi.org\/10.1088\/1742-6596\/898\/8\/082011","journal-title":"J Phys Conf Ser"},{"key":"74_CR29","unstructured":"HEP-SPEC Container Orchestrator (2021). https:\/\/gitlab.cern.ch\/hep-benchmarks\/hep-spec. Accessed 11 Nov 2021"},{"key":"74_CR30","unstructured":"ActiveMQ (2021). https:\/\/activemq.apache.org. Accessed 11 Nov 2021"},{"key":"74_CR31","doi-asserted-by":"publisher","first-page":"05012","DOI":"10.1051\/epjconf\/202024505012","volume":"245","author":"A Venkitesh","year":"2020","unstructured":"Venkitesh A et al (2020) Optimization of software on high performance computing platforms for the LUX-ZEPLIN dark matter experiment. EPJ Web Conf 245:05012. https:\/\/doi.org\/10.1051\/epjconf\/202024505012","journal-title":"EPJ Web Conf"},{"key":"74_CR32","doi-asserted-by":"publisher","unstructured":"De Maria R et al (2019) SixTrack Version 5: status and new developments. In: Proceedings of IPAC 2019. Melbourne, Australia: JACoW, 2019, pp 3200\u20133203. https:\/\/doi.org\/10.18429\/JACoW-IPAC2019-WEPTS043","DOI":"10.18429\/JACoW-IPAC2019-WEPTS043"},{"key":"74_CR33","doi-asserted-by":"publisher","first-page":"03045","DOI":"10.1051\/epjconf\/202125103045","volume":"251","author":"A Valassi","year":"2021","unstructured":"Valassi A et al (2021) Design and engineering of a simplified workflow execution for the MG5aMC event generator on GPUs and vector CPUs. EPJ Web Conf 251:03045. https:\/\/doi.org\/10.1051\/epjconf\/202125103045","journal-title":"EPJ Web Conf"},{"key":"74_CR34","doi-asserted-by":"publisher","first-page":"05009","DOI":"10.1051\/epjconf\/202024505009","volume":"245","author":"A Bocci","year":"2020","unstructured":"Bocci A et al (2020) Bringing heterogeneity to the CMS software framework. EPJ Web Conf 245:05009. https:\/\/doi.org\/10.1051\/epjconf\/202024505009","journal-title":"EPJ Web Conf"}],"container-title":["Computing and Software for Big Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-021-00074-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41781-021-00074-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-021-00074-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,26]],"date-time":"2021-12-26T11:07:32Z","timestamp":1640516852000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41781-021-00074-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":34,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["74"],"URL":"https:\/\/doi.org\/10.1007\/s41781-021-00074-y","relation":{},"ISSN":["2510-2036","2510-2044"],"issn-type":[{"type":"print","value":"2510-2036"},{"type":"electronic","value":"2510-2044"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"30 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 November 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 December 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"28"}}