{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T12:47:12Z","timestamp":1754398032855},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2018,12,29]],"date-time":"2018-12-29T00:00:00Z","timestamp":1546041600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Front. Comput. Sci."],"published-print":{"date-parts":[[2020,2]]},"DOI":"10.1007\/s11704-018-7319-2","type":"journal-article","created":{"date-parts":[[2018,12,29]],"date-time":"2018-12-29T02:20:20Z","timestamp":1546050020000},"page":"21-41","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Computer comparisons in the presence of performance variation"],"prefix":"10.1007","volume":"14","author":[{"given":"Samuel","family":"Irving","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bin","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaoming","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lu","family":"Peng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weihua","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lide","family":"Duan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2018,12,29]]},"reference":[{"key":"7319_CR1","first-page":"7","volume-title":"Proceedings of the 9th^IEEE International Symposium on High Performance Computer Architecture","author":"A R Alameldeen","year":"2003","unstructured":"Alameldeen A R, Wood D A. Variability in architectural simulations of multi-threaded workloads. In: Proceedings of the 9th IEEE International Symposium on High Performance Computer Architecture. 2003, 7\u201318"},{"issue":"10","key":"7319_CR2","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1145\/1297105.1297033","volume":"42","author":"A George","year":"2007","unstructured":"George A, Buytaer D, Eeckhout L. Statistically rigorous java performance evaluation. ACM SIGPLAN Notices, 2007, 42(10): 57\u201376","journal-title":"ACM SIGPLAN Notices"},{"key":"7319_CR3","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1145\/1508244.1508275","volume-title":"Proceedings of^ACM International Conference on Architectural Support for Programming Languages and Operating Systems","author":"T Mytkowicz","year":"2009","unstructured":"Mytkowicz T, Diwan A, Hauswirth M, Sweeney P F. Producing wrong data without doing anything obviously wrong. In: Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 2009, 265\u2013276"},{"issue":"3","key":"7319_CR4","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1145\/2658987","volume":"58","author":"S Krishnamurthi","year":"2015","unstructured":"Krishnamurthi S, Vitek J. The real software crisis: repeatability as a core value. Communications of ACM, 2015, 58(3): 34\u201336","journal-title":"Communications of ACM"},{"issue":"5","key":"7319_CR5","doi-asserted-by":"publisher","first-page":"1442","DOI":"10.1109\/TC.2014.2315614","volume":"64","author":"T Chen","year":"2015","unstructured":"Chen T, Guo Q, Temam O, Wu Y, Bao Y, Xu Z, Chen Y. Statistical performance comparisons of computers. IEEE Transactions on Computers, 2015, 64(5): 1442\u20131455","journal-title":"IEEE Transactions on Computers"},{"key":"7319_CR6","volume-title":"Statistical Methods. 3rd ed","author":"R J Freund","year":"2010","unstructured":"Freund R J, Mohr D,Wilson WJ. Statistical Methods. 3rd ed. London: Academic Press, 2010"},{"key":"7319_CR7","first-page":"1","volume-title":"Proceedings of the 18th^IEEE International Symposium On High Performance Computer Architecture","author":"T Chen","year":"2012","unstructured":"Chen T, Chen Y, Guo Q, Temam O, Wu Y, Hu W. Statistical performance comparisons of computers. In: Proceedings of the 18th IEEE International Symposium On High Performance Computer Architecture. 2012, 1\u201312"},{"key":"7319_CR8","volume-title":"Nonparametric Statistical Methods. 2nd ed","author":"M Hollander","year":"1999","unstructured":"Hollander M, Wolfe D A. Nonparametric Statistical Methods. 2nd ed. New York: John Wiley & Sons, 1999"},{"key":"7319_CR9","volume-title":"Introduction to the Practice of Statistics. 7th ed","author":"D Moore","year":"2010","unstructured":"Moore D, McCabe G P, Craig B. Introduction to the Practice of Statistics. 7th ed. New York: W. H. Freeman Press, 2010"},{"key":"7319_CR10","volume-title":"Randomization Tests. 3rd ed","author":"E S Edgington","year":"1995","unstructured":"Edgington E S. Randomization Tests. 3rd ed. New York: Marcel- Dekker, 1995"},{"key":"7319_CR11","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511802843","volume-title":"Bootstrap Methods and Their Application","author":"A C Davison","year":"1997","unstructured":"Davison A C, Hinkley D V. Bootstrap Methods and Their Application. New York: Cambridge University Press, 1997"},{"key":"7319_CR12","first-page":"488","volume-title":"Proceedings of the 20th^IEEE International Symposium on High-Performance Computer Architecture","author":"L Wang","year":"2014","unstructured":"Wang L, Zhan J, Luo C, Zhu Y, Yang Q, He Y. Bigdatabench: a big data benchmark suite from internet services. In: Proceedings of the 20th IEEE International Symposium on High-Performance Computer Architecture. 2014, 488\u2013499"},{"key":"7319_CR13","doi-asserted-by":"publisher","DOI":"10.1002\/9780470973196","volume-title":"Understanding Biplots","author":"J C Gower","year":"2011","unstructured":"Gower J C, Lubbe S G, Roux N L. Understanding Biplots. Hoboken: John Wiley & Sons, 2011"},{"key":"7319_CR14","doi-asserted-by":"crossref","DOI":"10.1201\/9780429246593","volume-title":"An Introduction to the Bootstrap","author":"B Efron","year":"1994","unstructured":"Efron B, Tibshirani R J. An Introduction to the Bootstrap. New York: Chapman and Hall\/CRC, 1994"},{"issue":"3","key":"7319_CR15","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1145\/5666.5673","volume":"29","author":"P J Fleming","year":"1986","unstructured":"Fleming P J, Wallace J J. How not to lie with statistics: the correct way to summarize benchmark results. Communications of the ACM, 1986, 29(3): 218\u2013221","journal-title":"Communications of the ACM"},{"key":"7319_CR16","volume-title":"Statistics: Principles and Methods. 6th ed","author":"R A Johnson","year":"2009","unstructured":"Johnson R A. Statistics: Principles and Methods. 6th ed. New York: John Wiley & Sons, 2009"},{"key":"7319_CR17","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1145\/1454115.1454128","volume-title":"Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques","author":"C Bienia","year":"2008","unstructured":"Bienia C, Kumar S, Singh J P, Li K. The PARSEC benchmark suite: characterization and architectural implications. In: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques. 2008, 72\u201381"},{"issue":"4","key":"7319_CR18","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1145\/1186736.1186738","volume":"34","author":"D Citron","year":"2006","unstructured":"Citron D, Hurani A, Gnadrey A. The harmonic or geometric mean: does it really matter? ACM SIGARCH Computer Architecture News, 2006, 34(4): 18\u201325","journal-title":"ACM SIGARCH Computer Architecture News"},{"key":"7319_CR19","first-page":"1","volume-title":"Proceedings of the 6th International Workshop on Unique chips and Systems","author":"M F Iqbal","year":"2010","unstructured":"Iqbal M F, John L K. Confusion by all means. In: Proceedings of the 6th International Workshop on Unique chips and Systems. 2010, 1\u20136"},{"issue":"4","key":"7319_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1040136.1040137","volume":"32","author":"J R Mashey","year":"2004","unstructured":"Mashey J R. War of the benchmark means: time for a truce. ACM SIGARCH Computer Architecture News, 2004, 32(4): 1\u201314","journal-title":"ACM SIGARCH Computer Architecture News"},{"key":"7319_CR21","volume-title":"Computer Architecture:^A Quantitative Approach. 4th ed","author":"J L Hennessy","year":"2007","unstructured":"Hennessy J L, Patterson D A. Computer Architecture: A Quantitative Approach. 4th ed. Walthan: Morgan Kaufmann, 2007"},{"key":"7319_CR22","doi-asserted-by":"publisher","DOI":"10.2200\/S00273ED1V01Y201006CAC010","volume-title":"Computer Architecture Performance Evaluation Methods","author":"L Eeckhout","year":"2010","unstructured":"Eeckhout L. Computer Architecture Performance Evaluation Methods. California: Morgan & Claypool Press, 2010"},{"key":"7319_CR23","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511612398","volume-title":"Measuring Computer Performance:^A Practitioner\u2019s Guide","author":"D J Lilja","year":"2000","unstructured":"Lilja D J. Measuring Computer Performance: A Practitioner\u2019s Guide. New York: Cambridge University Press, 2000"},{"key":"7319_CR24","first-page":"207","volume-title":"Proceedings of^ACM International Conference on Architectural Support for Programming Languages and Operating Systems","author":"A Oliveira","year":"2013","unstructured":"Oliveira A, Fischmeister S, Diwan A, Hauswirth M, Sweeney P F. Why you should care about quantile regression. In: Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 2013, 207\u2013218"},{"issue":"1","key":"7319_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/L-CA.2010.1","volume":"9","author":"S Patil","year":"2010","unstructured":"Patil S, Lilja D J. Using resampling techniques to compute confidence intervals for the harmonic mean of rate-based performance metrics. IEEE Computer Architecture Letters, 2010, 9(1): 1\u20134","journal-title":"IEEE Computer Architecture Letters"},{"key":"7319_CR26","first-page":"104","volume-title":"Proceedings of IEEE\/ACMInternational Symposium on Cluster, Cloud and Grid Computing, Newport Beach","author":"A Iosup","year":"2011","unstructured":"Iosup A, Yigitbasi N, Epema D H J. On the performance variability of production cloud services. In: Proceedings of IEEE\/ACMInternational Symposium on Cluster, Cloud and Grid Computing, Newport Beach. 2011, 104\u2013113"},{"issue":"3","key":"7319_CR27","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1145\/2885497","volume":"16","author":"P Leitner","year":"2016","unstructured":"Leitner P, Cito J. Patterns in the chaos\u2014a study of performance variation and predictability in public IaaS clouds. ACM Transactions on Internet Technology, 2016, 16(3): 15","journal-title":"ACM Transactions on Internet Technology"},{"key":"7319_CR28","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1109\/TPDS.2016.2613524","volume":"28","author":"W Zhang","year":"2016","unstructured":"Zhang W, Ji X, Song B, Yu S, Chen H, Li T, Yew P, Zhao W. Varcatcher: a pramework for tackling performance variability of parallel workloads on multi-core. IEEE Transactions on Parallel and Distributed Systems, 2016, 28: 1215\u20131228","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"issue":"4","key":"7319_CR29","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1145\/2086696.2086725","volume":"8","author":"K K Pusukuri","year":"2012","unstructured":"Pusukuri K K, Gupta R, Bhuyan A N. Thread tranquilizer: dynamically reducing performance variation. ACM Transactions on Architecture and Code Optimization, 2012, 8(4): 46\u201366","journal-title":"ACM Transactions on Architecture and Code Optimization"},{"key":"7319_CR30","first-page":"1077","volume-title":"Proceedings of the 1st^IEEE International Workshop on Variability in Parallel and Distributed Systems","author":"I Jimenez","year":"2016","unstructured":"Jimenez I, Maltzahn C, Lofstead J, Moody A, Mohror K, Arpaci-Dusseau R, Arpaci-Dusseau A. Characterizing and reducing cross-platform performance variability using OS-level virtualization. In: Proceedings of the 1st IEEE International Workshop on Variability in Parallel and Distributed Systems. 2016, 1077\u20131080"}],"container-title":["Frontiers of Computer Science"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11704-018-7319-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11704-018-7319-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11704-018-7319-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T07:01:56Z","timestamp":1662706916000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11704-018-7319-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,29]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,2]]}},"alternative-id":["7319"],"URL":"https:\/\/doi.org\/10.1007\/s11704-018-7319-2","relation":{},"ISSN":["2095-2228","2095-2236"],"issn-type":[{"value":"2095-2228","type":"print"},{"value":"2095-2236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,12,29]]},"assertion":[{"value":"9 September 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 January 2018","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 December 2018","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}