{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T18:33:54Z","timestamp":1771698834205,"version":"3.50.1"},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2014,10,27]],"date-time":"2014-10-27T00:00:00Z","timestamp":1414368000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2014,10,27]]},"abstract":"<jats:p>Running multiple programs on a processor aims at increasing the throughput of that processor. However, defining meaningful throughput metrics in a simulation environment is not as straightforward as reporting execution time. This has led to an ongoing debate on what forms a meaningful throughput metric for multiprogram workloads. We present a method to construct throughput metrics in a systematic way: we start by expressing assumptions on job size, job distribution, scheduling, and so forth that together define a theoretical throughput experiment. The throughput metric is then the average throughput of this experiment. Different assumptions lead to different metrics, so one should be aware of these assumptions when making conclusions based on results using a specific metric.<\/jats:p>\n          <jats:p>Throughput metrics should always be defined from explicit assumptions, because this leads to a better understanding of the implications and limits of the results obtained with that metric. We elaborate multiple metrics based on different assumptions. In particular, we identify the assumptions that lead to the commonly used weighted speedup and harmonic mean of speedups. Our study clarifies that they are actual throughput metrics, which was recently questioned. We also propose some new throughput metrics, which cannot always be expressed as a closed formula. We use real experimental data to characterize metrics and show how they relate to each other.<\/jats:p>","DOI":"10.1145\/2663346","type":"journal-article","created":{"date-parts":[[2014,10,28]],"date-time":"2014-10-28T12:40:29Z","timestamp":1414500029000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Multiprogram Throughput Metrics"],"prefix":"10.1145","volume":"11","author":[{"given":"Stijn","family":"Eyerman","sequence":"first","affiliation":[{"name":"Ghent University, Gent, Belgium"}]},{"given":"Pierre","family":"Michaud","sequence":"additional","affiliation":[{"name":"INRIA Rennes, France"}]},{"given":"Wouter","family":"Rogiest","sequence":"additional","affiliation":[{"name":"Ghent University, Gent, Belgium"}]}],"member":"320","published-online":{"date-parts":[[2014,10,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2006.73"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063384.2063454"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201914)","author":"Carlson T. E.","unstructured":"T. E. Carlson , W. Heirman , K. Van Craeynest , and L. Eeckhout . 2014. BarrierPoint: Sampled simulation of multi-threaded applications . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201914) . T. E. Carlson, W. Heirman, K. Van Craeynest, and L. Eeckhout. 2014. BarrierPoint: Sampled simulation of multi-threaded applications. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201914)."},{"key":"e_1_2_1_4_1","unstructured":"A. Carlton. 1995. CINT92 and CFP92 Homogeneous Capacity Method Offers Fair Measure of Processing Capacity. Retrieved from http:\/\/www.spec.org\/cpu92\/specrate.txt.  A. Carlton. 1995. CINT92 and CFP92 Homogeneous Capacity Method Offers Fair Measure of Processing Capacity. Retrieved from http:\/\/www.spec.org\/cpu92\/specrate.txt."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2540708.2540737"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2008.44"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736033"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"S. Eyerman and L. Eeckhout. 2013. Restating the case for weighted-IPC metrics to evaluate multiprogram workload performance. IEEE Computer Architecture Letters (April 2013).  S. Eyerman and L. Eeckhout. 2013. Restating the case for weighted-IPC metrics to evaluate multiprogram workload performance. IEEE Computer Architecture Letters (April 2013).","DOI":"10.1109\/L-CA.2013.9"},{"key":"e_1_2_1_9_1","volume-title":"MLP yes&excl","author":"Glew A.","unstructured":"A. Glew . 1998. MLP yes&excl ; ILP no&excl;. In ASPLOS Wild and Crazy Idea Session . 26--34. A. Glew. 1998. MLP yes&excl; ILP no&excl;. In ASPLOS Wild and Crazy Idea Session. 26--34."},{"key":"e_1_2_1_10_1","volume-title":"Workshop on Modeling, Benchmarking and Simulation (MoBS).","author":"Hilton A.","unstructured":"A. Hilton , N. Eswaran , and A. Roth . 2009. FIESTA: A sample-balanced multi-program workload methodology . In Workshop on Modeling, Benchmarking and Simulation (MoBS). A. Hilton, N. Eswaran, and A. Roth. 2009. FIESTA: A sample-balanced multi-program workload methodology. In Workshop on Modeling, Benchmarking and Simulation (MoBS)."},{"key":"e_1_2_1_11_1","volume-title":"Aggregating performance metrics over a benchmark suite","author":"John L. K.","unstructured":"L. K. John . 2006. Aggregating performance metrics over a benchmark suite . In Performance Evaluation and Benchmarking, L. K. John and L. Eeckhout (Eds.). CRC Press , 47--58. L. K. John. 2006. Aggregating performance metrics over a benchmark suite. In Performance Evaluation and Benchmarking, L. K. John and L. Eeckhout (Eds.). CRC Press, 47--58."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Workshop on Modeling, Benchmarking and Simulation (MoBS\u201905)","author":"Kihm J. L.","unstructured":"J. L. Kihm , T. Moseley , and D. A. Connors . 2005. A mathematical model for accurately balancing co-phase effects in simulated multithreaded programs . In Proceedings of the Workshop on Modeling, Benchmarking and Simulation (MoBS\u201905) . J. L. Kihm, T. Moseley, and D. A. Connors. 2005. A mathematical model for accurately balancing co-phase effects in simulated multithreaded programs. In Proceedings of the Workshop on Modeling, Benchmarking and Simulation (MoBS\u201905)."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the International Symposium on Computer Architecture (ISCA\u201904)","author":"Kumar R.","unstructured":"R. Kumar , D. M. Tullsen , P. Ranganathan , N. P. Jouppi , and K. I. Farkas . 2004. Single-ISA heterogeneous multi-core architectures for multithreaded workload performance . In Proceedings of the International Symposium on Computer Architecture (ISCA\u201904) . 81--92. R. Kumar, D. M. Tullsen, P. Ranganathan, N. P. Jouppi, and K. I. Farkas. 2004. Single-ISA heterogeneous multi-core architectures for multithreaded workload performance. In Proceedings of the International Symposium on Computer Architecture (ISCA\u201904). 81--92."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901)","author":"Luo K.","unstructured":"K. Luo , J. Gummaraju , and M. Franklin . 2001. Balancing throughput and fairness in SMT processors . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901) . 164--171. K. Luo, J. Gummaraju, and M. Franklin. 2001. Balancing throughput and fairness in SMT processors. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901). 164--171."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/L-CA.2012.25"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2009.5306795"},{"key":"e_1_2_1_18_1","volume-title":"Technical Report UW-CSE-00-04-02. University of Washington.","author":"Parekh S.","year":"2000","unstructured":"S. Parekh , S. Eggers , H. Levy , and J. Lo . 2000 . Thread-Sensitive Scheduling for SMT Processors . Technical Report UW-CSE-00-04-02. University of Washington. S. Parekh, S. Eggers, H. Levy, and J. Lo. 2000. Thread-Sensitive Scheduling for SMT Processors. Technical Report UW-CSE-00-04-02. University of Washington."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"M. Pinsky and S. Karlin. 2010. An Introduction to Stochastic Modeling. Academic Press.  M. Pinsky and S. Karlin. 2010. An Introduction to Stochastic Modeling. Academic Press.","DOI":"10.1016\/B978-0-12-381416-6.00001-0"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201903)","author":"Raasch S. E.","unstructured":"S. E. Raasch and S. K. Reinhardt . 2003. The impact of resource partitioning on SMT processors . In Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201903) . 15--26. S. E. Raasch and S. K. Reinhardt. 2003. The impact of resource partitioning on SMT processors. In Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201903). 15--26."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901)","author":"Sazeides Y.","unstructured":"Y. Sazeides and T. Juan . 2001. How to compare the performance of two SMT microarchitectures . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901) . 180--183. Y. Sazeides and T. Juan. 2001. How to compare the performance of two SMT microarchitectures. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201901). 180--183."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/605397.605403"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/378993.379244"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the International Conference on Parallel Architecture and Compilation Techniques (PACT\u201903)","author":"Tuck N.","unstructured":"N. Tuck and D. M. Tullsen . 2003. Initial observations of the simultaneous multithreading pentium 4 processor . In Proceedings of the International Conference on Parallel Architecture and Compilation Techniques (PACT\u201903) . 26--34. N. Tuck and D. M. Tullsen. 2003. Initial observations of the simultaneous multithreading pentium 4 processor. In Proceedings of the International Conference on Parallel Architecture and Compilation Techniques (PACT\u201903). 26--34."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 34th Annual IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201901)","author":"Tullsen D. M.","unstructured":"D. M. Tullsen and J. A. Brown . 2001. Handling long-latency loads in a simultaneous multithreading processor . In Proceedings of the 34th Annual IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201901) . 318--327. D. M. Tullsen and J. A. Brown. 2001. Handling long-latency loads in a simultaneous multithreading processor. In Proceedings of the 34th Annual IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201901). 318--327."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/232973.232993"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/223982.224449"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201906)","author":"Van Biesbrouck M.","unstructured":"M. Van Biesbrouck , L. Eeckhout , and B. Calder . 2006. Considering all starting points for simultaneous multithreading simulation . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201906) . 143--153. M. Van Biesbrouck, L. Eeckhout, and B. Calder. 2006. Considering all starting points for simultaneous multithreading simulation. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201906). 143--153."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2007.4362195"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201904)","author":"Van Biesbrouck M.","unstructured":"M. Van Biesbrouck , T. Sherwood , and B. Calder . 2004. A co-phase matrix to guide simultaneous multithreading simulation . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201904) . 45--56. M. Van Biesbrouck, T. Sherwood, and B. Calder. 2004. A co-phase matrix to guide simultaneous multithreading simulation. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201904). 45--56."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2011.6114194"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/L-CA.2011.1"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201913)","author":"Vel\u00e1squez R. A.","unstructured":"R. A. Vel\u00e1squez , P. Michaud , and A. Seznec . 2013. Selecting benchmark combinations for the evaluation of multicore throughput . In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201913) . 173--182. R. A. Vel\u00e1squez, P. Michaud, and A. Seznec. 2013. Selecting benchmark combinations for the evaluation of multicore throughput. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS\u201913). 173--182."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/1299042.1299057"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2663346","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2663346","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:12:47Z","timestamp":1750227167000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2663346"}},"subtitle":["A Systematic Approach"],"short-title":[],"issued":{"date-parts":[[2014,10,27]]},"references-count":33,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2014,10,27]]}},"alternative-id":["10.1145\/2663346"],"URL":"https:\/\/doi.org\/10.1145\/2663346","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,10,27]]},"assertion":[{"value":"2014-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-10-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}