{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T10:56:15Z","timestamp":1753440975407,"version":"3.37.3"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2021,1,2]],"date-time":"2021-01-02T00:00:00Z","timestamp":1609545600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,1,2]],"date-time":"2021-01-02T00:00:00Z","timestamp":1609545600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Supercomput"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Because of the increasing complexities of systems and applications, the performance of many traditional HPC benchmarks, such as HPL or HPCG, no longer correlates strongly with the actual performance of real applications. To address the discrepancy between simple benchmarks and real applications, and to better understand the application performance of systems, some metrics use a set of either real applications or mini applications. In particular, the Sustained System Performance (SSP) metric Kramer et al. (The NERSC sustained system performance (SSP) metric. Tech Rep LBNL-58868, 2005), which indicates the expected throughput of different applications executing with different datasets, is widely used. Whereas such a metric should lead to direct insights on the actual performance of real applications, sometimes more effort is necessary to port and evaluate complex applications. In this study, to obtain the approximate performance of SSP representing real applications, without running real applications, we propose a metric called the Simplified Sustained System Performance (SSSP) metric, which is computed based on several benchmark scores and their respective weighting factors, and we construct a method evaluating the SSSP metric of a system. The weighting factors are obtained by minimizing the gap between the SSP and SSSP scores based on a small set of reference systems. We evaluated the applicability of the SSSP method using eight systems and demonstrated that our proposed SSSP metrics produce appropriate performance projections of the SSP metrics of these systems, even when we adopted a simple method for computing the weighting factors. Additionally, the robustness of our SSSP metric was confirmed via computation of the weighting factors based on a smaller set of reference systems and computation of the SSSP metrics of other systems.<\/jats:p>","DOI":"10.1007\/s11227-020-03545-y","type":"journal-article","created":{"date-parts":[[2021,1,2]],"date-time":"2021-01-02T21:13:09Z","timestamp":1609621989000},"page":"6476-6504","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A new sustained system performance metric for scientific performance evaluation"],"prefix":"10.1007","volume":"77","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4709-1969","authenticated-orcid":false,"given":"Miwako","family":"Tsuji","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William T. C.","family":"Kramer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean-Christophe","family":"Weill","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean-Philippe","family":"Nomin\u00e9","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mitsuhisa","family":"Sato","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,1,2]]},"reference":[{"key":"3545_CR1","unstructured":"ECP proxy applications. https:\/\/proxyapps.exascaleproject.org\/"},{"key":"3545_CR2","unstructured":"HPC challenge. http:\/\/www.hpcchallenge.org\/"},{"key":"3545_CR3","unstructured":"Hpcg benchmark. http:\/\/www.hpcg-benchmark.org\/"},{"key":"3545_CR4","doi-asserted-by":"crossref","unstructured":"Aaziz O, Cook J, Cook J, Juedeman T, Richards D, Vaugha C (2018)A methodology for characterizing the correspondence between real and proxy applications. In: Proceedings of 2018 IEEE International Conference on Cluster Computing (CSTER). IEEE","DOI":"10.1109\/CLUSTER.2018.00037"},{"key":"3545_CR5","unstructured":"Armstrong B, Bae H, Eigenmann R, Saied F, Sayeed M, Zheng Y (2006) Hpc benchmarking and performance evaluation with realistic applications. In: The SPEC Benchmark Workshop, pp. 1\u201311"},{"key":"3545_CR6","unstructured":"Bailey D, Harris T, Saphir W, van der Wijngaart R, Woo A, Yarrow M (1995) The nas parallel benchmarks 2.0. Tech. Rep. NAS Technical Report NAS-95-020"},{"key":"3545_CR7","unstructured":"Bauer G, Hoefler T, Kramer W, Fiedler R (2012) Analyses and modeling of applications used to demonstrate sustained petascale performance on blue waters. In: Proceedings of the annual meeting of the cray users groups CUG-2012"},{"key":"3545_CR8","doi-asserted-by":"crossref","unstructured":"Boku T, Ishikawa K, Minami K, Nakamura Y, Shoji F, Takahasi D, Terai M, Ukawa A, Yosie T (2012) Multi-block\/multi-core SSOR preconditioner for the QCD quark solver for K computer. In: Proceedings of The 30th International Symposium on Lattice Field Theory","DOI":"10.22323\/1.164.0188"},{"key":"3545_CR9","unstructured":"Center for exascale simulation of combustion in turbulence: ExaCT. http:\/\/exactcodesign.org\/"},{"key":"3545_CR10","doi-asserted-by":"crossref","unstructured":"Change CH, Carpenter IL, Jones WB (2020)The ESIF-HPC-2 benchmark suite. In: Proceedings of the Principles and Practice of Parallel Programming (PPoPP) 2020","DOI":"10.1145\/3380868.3398200"},{"key":"3545_CR11","doi-asserted-by":"publisher","unstructured":"Danalis A, Marin G, McCurdy C, Meredith J, Roth P, Spafford K, Tipparaju V, Vetter J (2010) The scalable heterogeneous computing (shoc) benchmark suite. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems-ASPLOS . https:\/\/doi.org\/10.1145\/1735688.1735702","DOI":"10.1145\/1735688.1735702"},{"key":"3545_CR12","doi-asserted-by":"publisher","first-page":"1195","DOI":"10.1016\/S0167-8191(05)80033-X","volume":"17","author":"KM Dixit","year":"1991","unstructured":"Dixit KM (1991) The spec benchmarks. Parallel Computing 17:1195\u20131209","journal-title":"Parallel Computing"},{"key":"3545_CR13","doi-asserted-by":"crossref","unstructured":"Dongarra J, Heroux MA (2013) Toward a new metric for ranking high performance computing systems. Tech. Rep. SAND2013-4744, Sandia National Laboratories","DOI":"10.2172\/1089988"},{"key":"3545_CR14","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611971811","volume-title":"LINPACK user\u2019s guide","author":"J Dongarra","year":"1979","unstructured":"Dongarra J, Bunch CMJ, Stewart GW (1979) LINPACK user\u2019s guide. SIAM, Philadelphia"},{"key":"3545_CR15","unstructured":"Dongarra J, Luszczek P (2013) HPC challenge: design, history, and implementation highlights. In: Contemporary high performance computing: from petascale toward exascale. Taylor and Francis"},{"key":"3545_CR16","unstructured":"Ghosh S (2014) Distributed systems: an algorithmic approach, 2nd edn. Chapman and Hall\/CRC"},{"key":"3545_CR17","unstructured":"Heroux MA, Doerfler DW, Crozier PS, Willenbring JM, Edwards HC, Williams A, Rajan M, Keiter ER, Thorn-quist HK, Numrich RW (2009) Improving performance viamini-applications. Tech Rep SAND2009-5574, Sandia National Laboratories"},{"key":"3545_CR18","doi-asserted-by":"crossref","unstructured":"Karlin I, Bhatele A, Keasler J, Chamberlain BL, Cohen J, DeVito Z, Haque R, Laney D, Luke E, Wang F, Richards D, Schulz M, Still C (2013) Exploring traditional and emerging parallel programming models using a proxy application. In: 27th IEEE International Parallel and Distributed Processing Symposium (IEEE IPDPS 2013)","DOI":"10.1109\/IPDPS.2013.115"},{"key":"3545_CR19","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1007\/s11227-015-1616-4","volume":"72","author":"K Kononenko","year":"2016","unstructured":"Kononenko K (2016) An approach to error correction in program code using dynamic optimization in a virtual execution environment. J Supercomputing 72:845\u2013873. https:\/\/doi.org\/10.1007\/s11227-015-1616-4","journal-title":"J Supercomputing"},{"key":"3545_CR20","unstructured":"Kramer WTC (2013) Measuring sustained performance on blue waters with the SPP metric. In: Proceedings of the Annual Meeting of the Cray Users Groups CUG-2013"},{"key":"3545_CR21","unstructured":"Kramer WT (2008) Percu: a holistic method for evaluating high performance computing systems. Ph D thesis, EECS Department, University of California, Berkeley . http:\/\/www.eecs.berkeley.edu\/Pubs\/TechRpts\/2008\/EECS-2008-143.html"},{"key":"3545_CR22","doi-asserted-by":"crossref","unstructured":"Kramer WT, Shalf J, Strohmaier E (2005) The NERSC sustained system performance (SSP) metric. Tech Rep LBNL-58868","DOI":"10.2172\/861982"},{"key":"3545_CR23","unstructured":"Lawrence Livermore National Lab: LULESH. https:\/\/codesign.llnl.gov\/lulesh.php"},{"key":"3545_CR24","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511612398","volume-title":"Measuring computer performance a practitioner\u2019s guide","author":"DJ Lilja","year":"2000","unstructured":"Lilja DJ (2000) Measuring computer performance a practitioner\u2019s guide. Cambridge University Press, New York"},{"key":"3545_CR25","unstructured":"Maruyama N (2013) Mini-app effort in Japan. In: SC13 BoF: Library of mini-applications for exascale component-based performance modeling (Presentation)"},{"key":"3545_CR26","unstructured":"Maruyama N (2014) Miniapps for enabling architecture-application co-design for exascale supercomputing. In: 19th Workshop on sustained simulation performance (Presentation)"},{"key":"3545_CR27","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1016\/j.jocs.2015.03.007","volume":"10","author":"CL Mendes","year":"2015","unstructured":"Mendes CL, Bode B, Bauer GH, Enos J, Cristina Beldica WTK (2015) Deployment and testing of the sustained petascale blue waters system. J Comput Sci 10:327\u2013337","journal-title":"J Comput Sci"},{"key":"3545_CR28","unstructured":"NAS parallel benchmarks: http:\/\/www.nas.nasa.gov\/ publications\/npb.html"},{"key":"3545_CR29","unstructured":"Petitet A, Whaley RC, Dongarra J, Cleary A HPL-a portable implementation of the high-performance linpack benchmark for distributed-memory computers. http:\/\/www.netlib.org\/benchmark\/hpl\/"},{"key":"3545_CR30","unstructured":"Python Software for Convex Optimization. CVXOPT. https:\/\/cvxopt.org\/"},{"key":"3545_CR31","doi-asserted-by":"crossref","unstructured":"Richards D, Bhatele1 A, Aaziz O, Cook J, Finkel H, Homerding B, McCorquodale P, Mintz T, Moore S, Pavel R (2018) FY18 proxy app suite release, milestone report for the ecp proxy app project. Tech Rep LLNL-TR-760903, Exascale computing project","DOI":"10.2172\/1482870"},{"key":"3545_CR32","unstructured":"Sandia National Laboratories: Mantevo project. https:\/\/mantevo.org\/"},{"key":"3545_CR33","doi-asserted-by":"crossref","unstructured":"Sharkawi S, DeSota D, Panda R, Stevens S, Taylor V, Wu X (2012) SWAPP: a framework for performance projections of hpc applications using benchmarks. In: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops and PhD Forum, pp. 1722\u20131731. IEEE","DOI":"10.1109\/IPDPSW.2012.214"},{"key":"3545_CR34","doi-asserted-by":"crossref","unstructured":"Sharkawi S, DeSota D, Panda R, Indukuru R, Stevens S, Taylor V, Wu X (2009) Performance projection of hpc applications using spec cfp2006 benchmarks. In: Proceedings of the 23rd IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2009), pp. 1\u201312. IEEE","DOI":"10.1109\/IPDPS.2009.5161057"},{"issue":"10","key":"3545_CR35","doi-asserted-by":"publisher","first-page":"1202","DOI":"10.1145\/63039.63043","volume":"31","author":"JE Smith","year":"1988","unstructured":"Smith JE (1988) Characterizing computer performance with a single number. Commun ACM 31(10):1202\u20131206","journal-title":"Commun ACM"},{"key":"3545_CR36","unstructured":"Standard performance evaluation corporation: standard performance evaluation corporation. https:\/\/www.spec.org\/"},{"key":"3545_CR37","unstructured":"Terai M, Yashiro H, Sakamoto K, ichi Iga S, Tomita H, Satoh M, Minami K (2014) Performance optimization and evaluation of a global climate application using a 440m horizontal mesh on the K computer. In: Proceedings of 2014 International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2014)"},{"key":"3545_CR38","unstructured":"University of Illinois: Charm++:MiniApps. http:\/\/charmplusplus.org\/miniApps\/"},{"key":"3545_CR39","unstructured":"William TC (2013) Kramer: blue waters and the future of @scale computing and analysis, 4th AICS Workshop"}],"container-title":["The Journal of Supercomputing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11227-020-03545-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11227-020-03545-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11227-020-03545-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,21]],"date-time":"2021-06-21T10:21:03Z","timestamp":1624270863000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11227-020-03545-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,2]]},"references-count":39,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["3545"],"URL":"https:\/\/doi.org\/10.1007\/s11227-020-03545-y","relation":{},"ISSN":["0920-8542","1573-0484"],"issn-type":[{"type":"print","value":"0920-8542"},{"type":"electronic","value":"1573-0484"}],"subject":[],"published":{"date-parts":[[2021,1,2]]},"assertion":[{"value":"27 November 2020","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}