{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T14:15:34Z","timestamp":1761401734213,"version":"3.41.0"},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,1,20]],"date-time":"2015-01-20T00:00:00Z","timestamp":1421712000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2015,1,20]]},"abstract":"<jats:p>There are many compelling reasons to use a shared, public testbed such as GENI, Emulab, or PlanetLab to conduct experiments in computer science and networking. These testbeds support creating experiments with a large and diverse set of resources. Moreover these testbeds are constructed to inherently support the repeatability of experiments as required for scientifically sound research. Finally, the artifacts needed for a researcher to repeat their own experiment can be shared so that others can readily repeat the experiment in the same environment.<\/jats:p>\n          <jats:p>However using a shared, public testbed is different from conducting experiments on resources either owned by the experimenter or someone the experimenter knows. Experiments on shared, public testbeds are more likely to use large topologies, use scarce resources, and need to be tolerant to outages and maintenances in the testbed. In addition, experimenters may not have access to low-level debugging information.<\/jats:p>\n          <jats:p>This paper describes a methodology for new experimenters to write and deploy repeatable and sharable experiments which deal with these challenges by: having a clear plan; automating the execution and analysis of an experiment by following best practices from software engineering and system administration; and building scalable experiments. In addition, the paper describes a case study run on the GENI testbed which illustrates the methodology described.<\/jats:p>","DOI":"10.1145\/2723872.2723884","type":"journal-article","created":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T14:05:51Z","timestamp":1422453951000},"page":"90-99","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Creating Repeatable Computer Science and Networking Experiments on Shared, Public Testbeds"],"prefix":"10.1145","volume":"49","author":[{"given":"Sarah","family":"Edwards","sequence":"first","affiliation":[{"name":"GENI Project Office, Raytheon BBN Technologies, Cambridge, MA"}]},{"given":"Xuan","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Missouri, Kansas City, Kansas City, MO"}]},{"given":"Niky","family":"Riga","sequence":"additional","affiliation":[{"name":"GENI Project Office, Raytheon BBN Technologies, Cambridge, MA"}]}],"member":"320","published-online":{"date-parts":[[2015,1,20]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Adaptable profile-driven testbed. https:\/\/aptlab.net\/.  Adaptable profile-driven testbed. https:\/\/aptlab.net\/."},{"key":"e_1_2_1_2_1","unstructured":"Amazon web services website. http:\/\/aws.amazon.com.  Amazon web services website. http:\/\/aws.amazon.com."},{"key":"e_1_2_1_3_1","unstructured":"Bitbucket. https:\/\/bitbucket.org\/.  Bitbucket. https:\/\/bitbucket.org\/."},{"key":"e_1_2_1_4_1","unstructured":"Case study archive: Measuring ospf updates. http:\/\/groups.geni.net\/geni\/wiki\/PaperOSRMethodology.  Case study archive: Measuring ospf updates. http:\/\/groups.geni.net\/geni\/wiki\/PaperOSRMethodology."},{"key":"e_1_2_1_5_1","unstructured":"Github. https:\/\/github.com\/.  Github. https:\/\/github.com\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1007\/978-3-642-17851-1_31","volume-title":"Testbeds and Research Infrastructures. Development of Networks and Communities","author":"Albrecht J.","year":"2011"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2011.07.015"},{"key":"e_1_2_1_8_1","unstructured":"N. Bastin. geni-lib. http:\/\/geni-lib.readthedocs.org 2013-2014.  N. Bastin. geni-lib. http:\/\/geni-lib.readthedocs.org 2013-2014."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjp.2013.12.037"},{"volume-title":"ACM","year":"2007","author":"Chase J.","key":"e_1_2_1_10_1"},{"key":"e_1_2_1_11_1","unstructured":"F. Desprez G. Fox E. Jeannot K. Keahey M. Kozuch D. Margery P. Neyron L. Nussbaum C. Perez O. Richard W. Smith G. von Laszewski and J. Voeckler. Supporting experimental computer science. 03\/2012 2012.  F. Desprez G. Fox E. Jeannot K. Keahey M. Kozuch D. Margery P. Neyron L. Nussbaum C. Perez O. Richard W. Smith G. von Laszewski and J. Voeckler. Supporting experimental computer science. 03\/2012 2012."},{"key":"e_1_2_1_12_1","unstructured":"J. Duerig and etal Automatic ip address assignment on network topologies 2006.  J. Duerig and et al. Automatic ip address assignment on network topologies 2006."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723875"},{"volume-title":"Proceedings of the The Eleventh International Conference on Networks (ICN12)","year":"2012","author":"Fernandez M.","key":"e_1_2_1_15_1"},{"volume-title":"Inc.","year":"1991","author":"Jain R.","key":"e_1_2_1_16_1"},{"volume-title":"Indiana University","year":"2012","author":"Jonathan Klinginsmith J. Q.","key":"e_1_2_1_17_1"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2011.04.073"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2008.4630084"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2012.38"},{"volume-title":"Cambridge University Press","year":"2000","author":"Lilja D.","key":"e_1_2_1_21_1"},{"key":"e_1_2_1_22_1","first-page":"616","volume-title":"IM","author":"Liu X.","year":"2013"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"J. P. Mesirov. Computer science. accessible reproducible research. Science (New York N.Y.) 2010.  J. P. Mesirov. Computer science. accessible reproducible research. Science (New York N.Y.) 2010.","DOI":"10.1126\/science.1179653"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/THS.2010.5655108"},{"key":"e_1_2_1_25_1","first-page":"1","volume-title":"Proceedings of the 4th Conference on USENIX Symposium on Internet Technologies and Systems --","volume":"4","author":"Oppenheimer D.","year":"2003"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1113361.1113374"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1028788.1028824"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/774763.774772"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjp.2013.12.033"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1713254.1713267"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCNC.2005.1424763"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1201\/b16868"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(94)00111-Y"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/844128.844152"}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723884","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2723872.2723884","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:43Z","timestamp":1750227403000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723884"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,20]]},"references-count":33,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1,20]]}},"alternative-id":["10.1145\/2723872.2723884"],"URL":"https:\/\/doi.org\/10.1145\/2723872.2723884","relation":{},"ISSN":["0163-5980"],"issn-type":[{"type":"print","value":"0163-5980"}],"subject":[],"published":{"date-parts":[[2015,1,20]]},"assertion":[{"value":"2015-01-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}