{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:19:21Z","timestamp":1750306761280,"version":"3.41.0"},"reference-count":24,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2013,7,23]],"date-time":"2013-07-23T00:00:00Z","timestamp":1374537600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2013,7,23]]},"abstract":"<jats:p>Linux and other open-source Unix variants (and their distributors) provide researchers with full-fledged operating systems that are widely used. However, due to their complexity and rapid development, care should be exercised when using these operating systems for performance experiments, especially in systems research. In particular, the size and continual evolution of the Linux code-base makes it difficult to understand, and as a result, decipher and explain the reasons for performance improvements. In addition, the rapid kernel development cycle means that experimental results can be viewed as out of date, or meaningless, very quickly. We demonstrate that this viewpoint is incorrect because kernel changes can and have introduced both bugs and performance degradations.<\/jats:p>\n          <jats:p>This paper describes some of our experiences using Linux and FreeBSD as platforms for conducting performance evaluations and some performance regressions we have found. Our results show, these performance regressions can be serious (e.g., repeating identical experiments results in large variability in results) and long lived despite having a large negative effect on performance (one problem was present for more than 3 years). Based on these experiences, we argue: it is sometimes reasonable to use an older kernel version, experimental results need careful analysis to explain why a performance effect occurs, and publishing papers validating prior research is essential.<\/jats:p>","DOI":"10.1145\/2506164.2506175","type":"journal-article","created":{"date-parts":[[2013,7,25]],"date-time":"2013-07-25T19:12:41Z","timestamp":1374779561000},"page":"66-72","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Our troubles with Linux Kernel upgrades and why you should care"],"prefix":"10.1145","volume":"47","author":[{"given":"Ashif S.","family":"Harji","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}]},{"given":"Peter A.","family":"Buhr","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}]},{"given":"Tim","family":"Brecht","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}]}],"member":"320","published-online":{"date-parts":[[2013,7,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"August","author":"Baker M.","year":"2012","unstructured":"M. Baker . Independent labs to verify high-profile papers. Nature | News , August 2012 . M. Baker. Independent labs to verify high-profile papers. Nature | News, August 2012."},{"key":"e_1_2_1_2_1","volume-title":"February","author":"Blackburn S. M.","year":"2012","unstructured":"S. M. Blackburn , A. Diwan , M. Hauswirth , P. F. Sweeney , J. N. Amaral , V. Babka , W. Binder , T. Brecht , L. Bulej , L. Eeckhout , S. Fischmeister , D. Frampton , R. Garner , A. Georges , L. J. Hendren , M. Hind , A. L. Hosking , R. Jones , T. Kalibera , P. Moret , N. Nystrom , V. Pankratius , and P. Tuma . Evaluate collaboratory technical report #1: Can you trust your experimental results? , February 2012 . http:\/\/evaluate.inf.usi.ch\/technical-reports\/1. S. M. Blackburn, A. Diwan, M. Hauswirth, P. F. Sweeney, J. N. Amaral, V. Babka, W. Binder, T. Brecht, L. Bulej, L. Eeckhout, S. Fischmeister, D. Frampton, R. Garner, A. Georges, L. J. Hendren, M. Hind, A. L. Hosking, R. Jones, T. Kalibera, P. Moret, N. Nystrom, V. Pankratius, and P. Tuma. Evaluate collaboratory technical report #1: Can you trust your experimental results?, February 2012. http:\/\/evaluate.inf.usi.ch\/technical-reports\/1."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/1247415.1247417"},{"key":"e_1_2_1_4_1","unstructured":"CentOS: The community enterprise operating system. http:\/\/www.centos.org.  CentOS: The community enterprise operating system. http:\/\/www.centos.org."},{"key":"e_1_2_1_5_1","volume-title":"Mar.","author":"Corbet J.","year":"2012","unstructured":"J. Corbet , G. Kroah-Hartman , and A. McPherson . Linux kernel development: How fast it is going, who is doing it, what they are doing, and who is sponsoring it , Mar. 2012 . http:\/\/go.linuxfoundation.org\/who-writes-linux-2012. J. Corbet, G. Kroah-Hartman, and A. McPherson. Linux kernel development: How fast it is going, who is doing it, what they are doing, and who is sponsoring it, Mar. 2012. http:\/\/go.linuxfoundation.org\/who-writes-linux-2012."},{"key":"e_1_2_1_6_1","unstructured":"Evaluate Collaboratory. Experimental evaluation of software and systems in computer science. http:\/\/evaluate.inf.usi.ch\/.  Evaluate Collaboratory. Experimental evaluation of software and systems in computer science. http:\/\/evaluate.inf.usi.ch\/."},{"key":"e_1_2_1_7_1","volume-title":"December","author":"Feitelson D. G.","year":"2006","unstructured":"D. G. Feitelson . Experimental computer science: The need for a cultural change , December 2006 . http:\/\/www.cs.huji.ac.il\/ feit\/papers\/exp05.pdf. D. G. Feitelson. Experimental computer science: The need for a cultural change, December 2006. http:\/\/www.cs.huji.ac.il\/ feit\/papers\/exp05.pdf."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2103799.2103802"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2367589.2367591"},{"key":"e_1_2_1_11_1","volume-title":"Phoronix Media","author":"Larabel M.","year":"2010","unstructured":"M. Larabel . Five Years Of Linux Kernel Benchmarks: 2.6.12 Through 2.6.37 . Phoronix Media , Nov. 2010 . http:\/\/www.phoronix.com\/scan.php?page=article-&item=linux 2612 2637&num=1. M. Larabel. Five Years Of Linux Kernel Benchmarks: 2.6.12 Through 2.6.37. Phoronix Media, Nov. 2010. http:\/\/www.phoronix.com\/scan.php?page=article-&item=linux 2612 2637&num=1."},{"key":"e_1_2_1_12_1","volume-title":"The New Yorker","author":"Lehrer J.","year":"2010","unstructured":"J. Lehrer . The truth wears off: Is there something wrong with the scientific method ? The New Yorker , December 13, 2010 . J. Lehrer. The truth wears off: Is there something wrong with the scientific method? The New Yorker, December 13, 2010."},{"key":"e_1_2_1_13_1","unstructured":"Linux kernel performance! http:\/\/kernel-perf.sourceforge.net.  Linux kernel performance! http:\/\/kernel-perf.sourceforge.net."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/991130.991131"},{"key":"e_1_2_1_15_1","unstructured":"Netcraft Jan. 2012. https:\/\/ssl.netcraft.com\/sslsample-report\/CMatch\/oscnt all.  Netcraft Jan. 2012. https:\/\/ssl.netcraft.com\/sslsample-report\/CMatch\/oscnt all."},{"key":"e_1_2_1_16_1","unstructured":"OProfile 2012. http:\/\/oprofile.sourceforge.net.  OProfile 2012. http:\/\/oprofile.sourceforge.net."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1272996.1273021"},{"key":"e_1_2_1_18_1","volume-title":"Does systems research measure up? Technical report","author":"Small C.","year":"1997","unstructured":"C. Small , N. Ghosh , H. Saleeb , M. Seltzer , and K. Smith . Does systems research measure up? Technical report , Harvard University , TR-16-97, 1997 . C. Small, N. Ghosh, H. Saleeb, M. Seltzer, and K. Smith. Does systems research measure up? Technical report, Harvard University, TR-16-97, 1997."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2367589.2367602"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2229087.2229093"},{"key":"e_1_2_1_21_1","volume-title":"May","author":"SUSE","year":"2010","unstructured":"SUSE linux enterprise server for system z : The market leader for linux on ibm mainframes , May 2010 .http:\/\/www.novell.com\/docrep\/2010\/05\/-SLES for system z market share leader.pdf. SUSE linux enterprise server for system z: The market leader for linux on ibm mainframes, May 2010.http:\/\/www.novell.com\/docrep\/2010\/05\/-SLES for system z market share leader.pdf."},{"key":"e_1_2_1_22_1","unstructured":"The SystemTap home page 2010. http:\/\/sourceware.org\/systemtap\/.  The SystemTap home page 2010. http:\/\/sourceware.org\/systemtap\/."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.675631"},{"key":"e_1_2_1_24_1","unstructured":"Top 500 supercomputer sites. http:\/\/www.top500.org\/statistics\/list Category\u2192Operating System Family\u2192Submit.  Top 500 supercomputer sites. http:\/\/www.top500.org\/statistics\/list Category\u2192Operating System Family\u2192Submit."},{"key":"e_1_2_1_25_1","volume-title":"Aug.","author":"Surveys Web Technology","year":"2011","unstructured":"Web Technology Surveys . Historical trends in the usage of Linux versions for websites , Aug. 2011 . http:\/\/w3techs.com\/technologies\/history details\/oslinux. Web Technology Surveys. Historical trends in the usage of Linux versions for websites, Aug. 2011. http:\/\/w3techs.com\/technologies\/history details\/oslinux."}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2506164.2506175","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2506164.2506175","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:28:43Z","timestamp":1750231723000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2506164.2506175"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,7,23]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,7,23]]}},"alternative-id":["10.1145\/2506164.2506175"],"URL":"https:\/\/doi.org\/10.1145\/2506164.2506175","relation":{},"ISSN":["0163-5980"],"issn-type":[{"type":"print","value":"0163-5980"}],"subject":[],"published":{"date-parts":[[2013,7,23]]},"assertion":[{"value":"2013-07-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}