{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:16:37Z","timestamp":1750306597278,"version":"3.41.0"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,1,20]],"date-time":"2015-01-20T00:00:00Z","timestamp":1421712000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2015,1,20]]},"abstract":"<jats:p>Repeating research in computer science requires more than just code and data: it requires an appropriate environment in which to run experiments. In some cases, this environment appears fairly straightforward: it consists of a particular operating system and set of required libraries. In many cases, however, it is considerably more complex: the execution environment may be an entire network, may involve complex and fragile configuration of the dependencies, or may require large amounts of resources in terms of computation cycles, network bandwidth, or storage. Even the \"straightforward\" case turns out to be surprisingly intricate: there may be explicit or hidden dependencies on compilers, kernel quirks, details of the ISA, etc. The result is that when one tries to repeat published results, creating an environment sufficiently similar to one in which the experiment was originally run can be troublesome; this problem only gets worse as time passes. What the computer science community needs, then, are environments that have the explicit goal of enabling repeatable research. This paper outlines the problem of repeatable research environments, presents a set of requirements for such environments, and describes one facility that attempts to address them.<\/jats:p>","DOI":"10.1145\/2723872.2723885","type":"journal-article","created":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T14:05:51Z","timestamp":1422453951000},"page":"100-107","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Apt"],"prefix":"10.1145","volume":"49","author":[{"given":"Robert","family":"Ricci","sequence":"first","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Gary","family":"Wong","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Leigh","family":"Stoller","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Kirk","family":"Webb","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Jonathon","family":"Duerig","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Keith","family":"Downie","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]},{"given":"Mike","family":"Hibler","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah"}]}],"member":"320","published-online":{"date-parts":[[2015,1,20]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Sixth International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TridentCom)","author":"Albrecht J.","year":"2010","unstructured":"J. Albrecht and D. Y. Huang . Managing distributed applications using Gush . In Proceedings of the Sixth International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TridentCom) , May 2010 . J. Albrecht and D. Y. Huang. Managing distributed applications using Gush. In Proceedings of the Sixth International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TridentCom), May 2010."},{"doi-asserted-by":"publisher","key":"e_1_2_1_2_1","DOI":"10.1016\/j.bjp.2013.12.037"},{"key":"e_1_2_1_3_1","volume-title":"The 21st Large Installation System Administration Conference (LISA)","author":"Cappos J.","year":"2007","unstructured":"J. Cappos , S. Baker , J. Plichta , D. Nyugen , J. Hardies , M. Borgard , J. Johnston , and J. Hartman . Stork: Package management for distributed VM environments . In The 21st Large Installation System Administration Conference (LISA) , Nov. 2007 . J. Cappos, S. Baker, J. Plichta, D. Nyugen, J. Hardies, M. Borgard, J. Johnston, and J. Hartman. Stork: Package management for distributed VM environments. In The 21st Large Installation System Administration Conference (LISA), Nov. 2007."},{"key":"e_1_2_1_4_1","volume-title":"Measuring reproducibility in computer systems research. Technical report","author":"Collberg C.","year":"2013","unstructured":"C. Collberg , T. Proebsting , G. Moraila , A. Shankaran , Z. Shi , and A. M. Warren . Measuring reproducibility in computer systems research. Technical report , University of Arizona , Mar. 2013 . C. Collberg, T. Proebsting, G. Moraila, A. Shankaran, Z. Shi, and A. M. Warren. Measuring reproducibility in computer systems research. Technical report, University of Arizona, Mar. 2013."},{"unstructured":"Docker. https:\/\/www.docker.com\/.  Docker. https:\/\/www.docker.com\/.","key":"e_1_2_1_5_1"},{"unstructured":"The dpkg package manager. https:\/\/wiki.debian.org\/Teams\/Dpkg.  The dpkg package manager. https:\/\/wiki.debian.org\/Teams\/Dpkg.","key":"e_1_2_1_6_1"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation","author":"Eide E.","year":"2007","unstructured":"E. Eide , L. Stoller , and J. Lepreau . An experimentation workbench for replayable networking research . In Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation , Oct. 2007 . E. Eide, L. Stoller, and J. Lepreau. An experimentation workbench for replayable networking research. In Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation, Oct. 2007."},{"unstructured":"Elsevier Ltd. The executable paper grand challenge. http:\/\/www.executablepapers.com\/ 2011.  Elsevier Ltd. The executable paper grand challenge. http:\/\/www.executablepapers.com\/ 2011.","key":"e_1_2_1_8_1"},{"unstructured":"Examining \"Reproducibility in Computer Science\". http:\/\/cs.brown.edu\/~sk\/Memos\/ Examining-Reproducibility\/.  Examining \"Reproducibility in Computer Science\". http:\/\/cs.brown.edu\/~sk\/Memos\/ Examining-Reproducibility\/.","key":"e_1_2_1_9_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_10_1","DOI":"10.1016\/j.procs.2011.04.062"},{"doi-asserted-by":"publisher","key":"e_1_2_1_11_1","DOI":"10.1016\/j.bjp.2013.10.016"},{"doi-asserted-by":"publisher","key":"e_1_2_1_12_1","DOI":"10.1109\/MCSE.2012.36"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance. USENIX Association","author":"Guo P. J.","year":"2012","unstructured":"P. J. Guo and M. Seltzer . Burrito: Wrapping your lab notebook in computational infrastructure . In Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance. USENIX Association , 2012 . P. J. Guo and M. Seltzer. Burrito: Wrapping your lab notebook in computational infrastructure. In Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance. USENIX Association, 2012."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the USENIX Annual Technical Conference. USENIX","author":"Hibler M.","year":"2003","unstructured":"M. Hibler , L. Stoller , J. Lepreau , R. Ricci , and C. Barb . Fast, scalable disk imaging with Frisbee . In Proceedings of the USENIX Annual Technical Conference. USENIX , June 2003 . M. Hibler, L. Stoller, J. Lepreau, R. Ricci, and C. Barb. Fast, scalable disk imaging with Frisbee. In Proceedings of the USENIX Annual Technical Conference. USENIX, June 2003."},{"key":"e_1_2_1_15_1","volume-title":"Measurement, Simulation, and Modeling","author":"Jain R.","year":"1991","unstructured":"R. Jain . The Art of Computer Systems Performance Analysis: Techniques for Experimental Design , Measurement, Simulation, and Modeling . Wiley , 1991 . R. Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley, 1991."},{"unstructured":"Labwiki. http:\/\/labwiki.mytestbed.net\/.  Labwiki. http:\/\/labwiki.mytestbed.net\/.","key":"e_1_2_1_16_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_17_1","DOI":"10.1109\/INFCOM.2007.304"},{"doi-asserted-by":"publisher","key":"e_1_2_1_18_1","DOI":"10.1145\/1508244.1508275"},{"doi-asserted-by":"publisher","key":"e_1_2_1_19_1","DOI":"10.1016\/j.procs.2011.04.064"},{"unstructured":"ns-3. http:\/\/www.nsnam.org\/.  ns-3. http:\/\/www.nsnam.org\/.","key":"e_1_2_1_20_1"},{"unstructured":"OCCAM\n  : Open curation for computer architecture modeling. http:\/\/www.occamportal.org\/.  OCCAM: Open curation for computer architecture modeling. http:\/\/www.occamportal.org\/.","key":"e_1_2_1_21_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_22_1","DOI":"10.1145\/2479871.2479892"},{"unstructured":"Puppet. http:\/\/puppetlabs.com\/.  Puppet. http:\/\/puppetlabs.com\/.","key":"e_1_2_1_23_1"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the Conference on File and Storage Technologies (FAST)","author":"Qunlan S.","year":"2002","unstructured":"S. Qunlan and S. Dorward . Venti: A new approach to archival storage . In Proceedings of the Conference on File and Storage Technologies (FAST) , Jan. 2002 . S. Qunlan and S. Dorward. Venti: A new approach to archival storage. In Proceedings of the Conference on File and Storage Technologies (FAST), Jan. 2002."},{"key":"e_1_2_1_25_1","volume-title":"SOSP Workshop on Real Overlays and Distributed Systems (ROADS)","author":"Rakotoarivelo T.","year":"2009","unstructured":"T. Rakotoarivelo , M. Ott , I. Seskar , and G. Jourjon . OMF: a control and management framework for networking testbeds . In SOSP Workshop on Real Overlays and Distributed Systems (ROADS) , Oct. 2009 . T. Rakotoarivelo, M. Ott, I. Seskar, and G. Jourjon. OMF: a control and management framework for networking testbeds. In SOSP Workshop on Real Overlays and Distributed Systems (ROADS), Oct. 2009."},{"unstructured":"Resource specification (RSpec) documents in GENI. http:\/\/groups.geni.net\/geni\/wiki\/GENIExperimenter\/RSpecs.  Resource specification (RSpec) documents in GENI. http:\/\/groups.geni.net\/geni\/wiki\/GENIExperimenter\/RSpecs.","key":"e_1_2_1_26_1"},{"unstructured":"The RPM package manager. http:\/\/www.rpm.org.  The RPM package manager. http:\/\/www.rpm.org.","key":"e_1_2_1_27_1"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the DETER Community Workshop on Cyber Security Experimentation and Test","author":"Schwab S.","year":"2007","unstructured":"S. Schwab , B. Wilson , C. Ko , and A. Hussain . SEER: A security experimentation environment for DETER . In Proceedings of the DETER Community Workshop on Cyber Security Experimentation and Test , Aug. 2007 . S. Schwab, B. Wilson, C. Ko, and A. Hussain. SEER: A security experimentation environment for DETER. In Proceedings of the DETER Community Workshop on Cyber Security Experimentation and Test, Aug. 2007."},{"doi-asserted-by":"publisher","key":"e_1_2_1_29_1","DOI":"10.1145\/2038642.2038650"},{"doi-asserted-by":"publisher","key":"e_1_2_1_30_1","DOI":"10.5555\/1060289.1060313"},{"unstructured":"Wind River Systems. Simics full system simulator. http:\/\/www.windriver.com\/products\/simics\/.  Wind River Systems. Simics full system simulator. http:\/\/www.windriver.com\/products\/simics\/.","key":"e_1_2_1_31_1"}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723885","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2723872.2723885","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:43Z","timestamp":1750227403000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723885"}},"subtitle":["A Platform for Repeatable Research in Computer Science"],"short-title":[],"issued":{"date-parts":[[2015,1,20]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1,20]]}},"alternative-id":["10.1145\/2723872.2723885"],"URL":"https:\/\/doi.org\/10.1145\/2723872.2723885","relation":{},"ISSN":["0163-5980"],"issn-type":[{"type":"print","value":"0163-5980"}],"subject":[],"published":{"date-parts":[[2015,1,20]]},"assertion":[{"value":"2015-01-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}