{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T01:31:29Z","timestamp":1768008689650,"version":"3.49.0"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"4a","license":[{"start":{"date-parts":[[2012,3,18]],"date-time":"2012-03-18T00:00:00Z","timestamp":1332028800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGPLAN Not."],"published-print":{"date-parts":[[2012,6,18]]},"abstract":"<jats:p>Computer systems research spans sub-disciplines that include embedded systems, programming languages and compilers, networking, and operating systems. Our contention is that a number of structural factors inhibit quality systems research. We highlight some of the factors we have encountered in our own work and observed in published papers and propose solutions that could both increase the productivity of researchers and the quality of their output.<\/jats:p>","DOI":"10.1145\/2442776.2442781","type":"journal-article","created":{"date-parts":[[2013,2,22]],"date-time":"2013-02-22T19:25:04Z","timestamp":1361561104000},"page":"30-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["R3"],"prefix":"10.1145","volume":"47","author":[{"given":"Jan","family":"Vitek","sequence":"first","affiliation":[{"name":"Purdue University, USA"}]},{"given":"Tomas","family":"Kalibera","sequence":"additional","affiliation":[{"name":"University of Kent, UK"}]}],"member":"320","published-online":{"date-parts":[[2012,3,18]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Evaluate collaboratory: Experimental evaluation of software and systems in computer science. http:\/\/evaluate.inf.usi.ch\/ 2011.  Evaluate collaboratory: Experimental evaluation of software and systems in computer science. http:\/\/evaluate.inf.usi.ch\/ 2011."},{"key":"e_1_2_1_2_1","volume-title":"http:\/\/www.rrplanet.com\/","author":"Reproducible","year":"2011","unstructured":"Reproducible research planet. http:\/\/www.rrplanet.com\/ , 2011 . Reproducible research planet. http:\/\/www.rrplanet.com\/, 2011."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1147\/sj.391.0211"},{"key":"e_1_2_1_4_1","volume-title":"Annals of Applied Statistics","author":"Baggerly K.","year":"2008","unstructured":"K. Baggerly and K. Coombes . Deriving chemo sensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology . Annals of Applied Statistics , 2008 . K. Baggerly and K. Coombes. Deriving chemo sensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. Annals of Applied Statistics, 2008."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176324716"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1167473.1167488"},{"key":"e_1_2_1_7_1","first-page":"137","volume-title":"High Performance Garbage Collection in Java with MMTk. In Proceedings of the 26th International Conference on Software Engineering (ICSE)","author":"Blackburn S. M.","year":"2004","unstructured":"S. M. Blackburn , P. Cheng , and K. S. McKinley . Oil and Water ? High Performance Garbage Collection in Java with MMTk. In Proceedings of the 26th International Conference on Software Engineering (ICSE) , pages 137 -- 146 , 2004 . S. M. Blackburn, P. Cheng, and K. S. McKinley. Oil and Water? High Performance Garbage Collection in Java with MMTk. In Proceedings of the 26th International Conference on Software Engineering (ICSE), pages 137--146, 2004."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/1247415.1247462"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511802843"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1954.tb00159.x"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1542476.1542528"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297027.1297033"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1449764.1449794"},{"key":"e_1_2_1_14_1","volume-title":"Component And Middleware Performance Workshop, OOPSLA","author":"Gu D.","year":"2004","unstructured":"D. Gu , C. Verbrugge , and E. Gagnon . Code layout as a source of noise in JVM performance . In Component And Middleware Performance Workshop, OOPSLA , 2004 . D. Gu, C. Verbrugge, and E. Gagnon. Code layout as a source of noise in JVM performance. In Component And Middleware Performance Workshop, OOPSLA, 2004."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1712605.1712640"},{"key":"e_1_2_1_16_1","volume-title":"The Art of Computer Systems Performance Analysis","author":"Jain R.","year":"1991","unstructured":"R. Jain . The Art of Computer Systems Performance Analysis . Wiley , 1991 . R. Jain. The Art of Computer Systems Performance Analysis. Wiley, 1991."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/1097871.1098168"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.1677"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/11777830_5"},{"key":"e_1_2_1_20_1","volume-title":"Experimental Methods: An Introduction to the Analysis and Presentation of Data","author":"Kirkup L.","year":"1994","unstructured":"L. Kirkup . Experimental Methods: An Introduction to the Analysis and Presentation of Data . Wiley , 1994 . L. Kirkup. Experimental Methods: An Introduction to the Analysis and Presentation of Data. Wiley, 1994."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511612398","volume-title":"Measuring Computer Performance: A Practitioner's Guide","author":"Lilja D. J.","year":"2000","unstructured":"D. J. Lilja . Measuring Computer Performance: A Practitioner's Guide . Cambridge University Press , 2000 . D. J. Lilja. Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press, 2000."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/L-CA.2004.6"},{"key":"e_1_2_1_23_1","volume-title":"Designing experiments and analyzing data: a model comparison perspective","author":"Maxwell S. E.","year":"2004","unstructured":"S. E. Maxwell and H. D. Delaney . Designing experiments and analyzing data: a model comparison perspective . Routledge , 2004 . S. E. Maxwell and H. D. Delaney. Designing experiments and analyzing data: a model comparison perspective. Routledge, 2004."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1508244.1508275"},{"issue":"2","key":"e_1_2_1_25_1","first-page":"175","article-title":"The robustness of parametric statistical methods","volume":"46","author":"Rasch D.","year":"2004","unstructured":"D. Rasch and V. Guiard . The robustness of parametric statistical methods . Psychology Science , 46 ( 2 ): 175 -- 208 , 2004 . D. Rasch and V. Guiard. The robustness of parametric statistical methods. Psychology Science, 46(2):175--208, 2004.","journal-title":"Psychology Science"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2048066.2048119"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806596.1806598"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","DOI":"10.6028\/NIST.TN.1297","volume-title":"Guidelines for evaluating and expressing the uncertainty of NIST measurement results. Technical Note 1297","author":"Taylor B. N.","year":"1994","unstructured":"B. N. Taylor and C. E. Kuyatt . Guidelines for evaluating and expressing the uncertainty of NIST measurement results. Technical Note 1297 , National Institute of Standards and Technology , 1994 . B. N. Taylor and C. E. Kuyatt. Guidelines for evaluating and expressing the uncertainty of NIST measurement results. Technical Note 1297, National Institute of Standards and Technology, 1994."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1506409.1506410"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001269.2001283"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2009.17"},{"key":"e_1_2_1_32_1","volume-title":"An Introduction to Scientific Research","author":"Wilson E. B.","year":"1952","unstructured":"E. B. Wilson . An Introduction to Scientific Research . McGraw Hill , 1952 . E. B.Wilson. An Introduction to Scientific Research. McGraw Hill, 1952."}],"container-title":["ACM SIGPLAN Notices"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2442776.2442781","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2442776.2442781","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:18:39Z","timestamp":1750234719000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2442776.2442781"}},"subtitle":["repeatability, reproducibility and rigor"],"short-title":[],"issued":{"date-parts":[[2012,3,18]]},"references-count":32,"journal-issue":{"issue":"4a","published-print":{"date-parts":[[2012,6,18]]}},"alternative-id":["10.1145\/2442776.2442781"],"URL":"https:\/\/doi.org\/10.1145\/2442776.2442781","relation":{},"ISSN":["0362-1340","1558-1160"],"issn-type":[{"value":"0362-1340","type":"print"},{"value":"1558-1160","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,3,18]]},"assertion":[{"value":"2012-03-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}