{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T08:51:28Z","timestamp":1770281488560,"version":"3.49.0"},"reference-count":16,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T00:00:00Z","timestamp":1590105600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGCOMM Comput. Commun. Rev."],"published-print":{"date-parts":[[2020,5,22]]},"abstract":"<jats:p>Reproducibility: the extent to which consistent results are obtained when an experiment is repeated, is important as a means to validate experimental results, promote integrity of research, and accelerate follow up work. Commitment to artifact reviewing and badging seeks to promote reproducibility and rank the quality of submitted artifacts.<\/jats:p>\n          <jats:p>However, as illustrated in this issue, the current badging scheme, with its focus upon an artifact being reusable, may not identify limitations of architecture, implementation, or evaluation.<\/jats:p>\n          <jats:p>We propose that to improve the insight into artifact reproducibility, the depth and nature of artifact evaluation must move beyond simply considering if an artifact is reusable. Artifact evaluation should consider the methods of that evaluation alongside the varying of inputs to that evaluation. To achieve this, we suggest an extension to the scope of artifact badging, and describe both approaches and best practice arising in other communities. We seek to promote conversation and make a call to action intended to strengthen the scientific method within our domain.<\/jats:p>","DOI":"10.1145\/3402413.3402422","type":"journal-article","created":{"date-parts":[[2020,5,25]],"date-time":"2020-05-25T22:41:33Z","timestamp":1590446493000},"page":"60-63","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Thoughts about Artifact Badging"],"prefix":"10.1145","volume":"50","author":[{"given":"Noa","family":"Zilberman","sequence":"first","affiliation":[{"name":"University of Oxford"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew W.","family":"Moore","sequence":"additional","affiliation":[{"name":"University of Cambridge"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,5,23]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"ACM. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/ policies\/artifact-review-badging.  ACM. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/ policies\/artifact-review-badging."},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Vaibhav Bajpai Anna Brunstrom Anja Feldmann Wolfgang Kellerer Aiko Pras Henning Schulzrinne Georgios Smaragdakis Matthias W\u00e4hlisch and Klaus Wehrle. 2019. The Dagstuhl beginners guide to reproducibility for experimental networking research.  Vaibhav Bajpai Anna Brunstrom Anja Feldmann Wolfgang Kellerer Aiko Pras Henning Schulzrinne Georgios Smaragdakis Matthias W\u00e4hlisch and Klaus Wehrle. 2019. The Dagstuhl beginners guide to reproducibility for experimental networking research.","DOI":"10.1145\/3314212.3314217"},{"key":"e_1_2_1_3_1","unstructured":"Lorena Barba and Grigori Fursin. 2019. Reproducibility Initiative. https: \/\/sc19.supercomputing.org\/submit\/reproducibility-initiative\/.  Lorena Barba and Grigori Fursin. 2019. Reproducibility Initiative. https: \/\/sc19.supercomputing.org\/submit\/reproducibility-initiative\/."},{"key":"e_1_2_1_4_1","unstructured":"E. D. Berger S. M. Blackburn M. Hauswirth and M. W. Hicks. 2019. A Checklist Manifesto for Empirical Evaluation: A Preemptive Strike Against a Replication Crisis in Computer Science.  E. D. Berger S. M. Blackburn M. Hauswirth and M. W. Hicks. 2019. A Checklist Manifesto for Empirical Evaluation: A Preemptive Strike Against a Replication Crisis in Computer Science."},{"key":"e_1_2_1_5_1","unstructured":"E. D. Berger S. M. Blackburn M. Hauswirth and M. W. Hicks. 2019. Empirical Evaluation Guidelines.  E. D. Berger S. M. Blackburn M. Hauswirth and M. W. Hicks. 2019. Empirical Evaluation Guidelines."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994031"},{"key":"e_1_2_1_7_1","volume-title":"SIGCOMM","author":"Handley Mark","unstructured":"Mark Handley , Costin Raiciu , Alexandru Agache , Andrei Voinescu , Andrew W Moore , Gianni Antichi , and Marcin W\u00f3jcik . 2017. Re-architecting datacenter networks and stacks for low latency and high performance . In SIGCOMM . ACM , Los Angeles, CA, USA , 29--42. Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W Moore, Gianni Antichi, and Marcin W\u00f3jcik. 2017. Re-architecting datacenter networks and stacks for low latency and high performance. In SIGCOMM. ACM, Los Angeles, CA, USA, 29--42."},{"key":"e_1_2_1_8_1","volume-title":"Systems Benchmarking Crimes. https:\/\/www.cse.unsw.edu. au\/~gernot\/benchmarking-crimes.html [accessed","author":"Heiser Gernot","year":"2020","unstructured":"Gernot Heiser . 2020. Systems Benchmarking Crimes. https:\/\/www.cse.unsw.edu. au\/~gernot\/benchmarking-crimes.html [accessed Jan. 2020 ]. Gernot Heiser. 2020. Systems Benchmarking Crimes. https:\/\/www.cse.unsw.edu. au\/~gernot\/benchmarking-crimes.html [accessed Jan. 2020]."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2743015"},{"key":"e_1_2_1_10_1","unstructured":"Michael A. Heroux. 2018. SC Reproducibility Initiative. https:\/\/sc18. supercomputing.org\/submit\/sc-reproducibility-initiative\/index.html.  Michael A. Heroux. 2018. SC Reproducibility Initiative. https:\/\/sc18. supercomputing.org\/submit\/sc-reproducibility-initiative\/index.html."},{"key":"e_1_2_1_11_1","unstructured":"Beth Plale. 2020. Transparency and Reproducibility Initiative. https:\/\/sc20. supercomputing.org\/submit\/transparency-reproducibility-initiative\/.  Beth Plale. 2020. Transparency and Reproducibility Initiative. https:\/\/sc20. supercomputing.org\/submit\/transparency-reproducibility-initiative\/."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336937.3336944"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097766.3097768"},{"key":"e_1_2_1_14_1","volume-title":"Benchmarking Crimes: An Emerging Threat in Systems Security. arXiv:1801.02381 http:\/\/arxiv.org\/abs\/1801.02381","author":"van der Kouwe Erik","year":"2018","unstructured":"Erik van der Kouwe , Dennis Andriesse , Herbert Bos , Cristiano Giuffrida , and Gernot Heiser . 2018 . Benchmarking Crimes: An Emerging Threat in Systems Security. arXiv:1801.02381 http:\/\/arxiv.org\/abs\/1801.02381 Erik van der Kouwe, Dennis Andriesse, Herbert Bos, Cristiano Giuffrida, and Gernot Heiser. 2018. Benchmarking Crimes: An Emerging Threat in Systems Security. arXiv:1801.02381 http:\/\/arxiv.org\/abs\/1801.02381"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3402413.3402418"},{"key":"e_1_2_1_16_1","volume-title":"Stardust: Divide and Conquer in the Data Center Network. In NSDI.","author":"Zilberman Noa","year":"2019","unstructured":"Noa Zilberman , Gabi Bracha , and Golan Schzukin . 2019 . Stardust: Divide and Conquer in the Data Center Network. In NSDI. Boston, MA, USA , 141--160. Noa Zilberman, Gabi Bracha, and Golan Schzukin. 2019. Stardust: Divide and Conquer in the Data Center Network. In NSDI. Boston, MA, USA, 141--160."}],"container-title":["ACM SIGCOMM Computer Communication Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3402413.3402422","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3402413.3402422","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:34Z","timestamp":1750200094000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3402413.3402422"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,22]]},"references-count":16,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,5,22]]}},"alternative-id":["10.1145\/3402413.3402422"],"URL":"https:\/\/doi.org\/10.1145\/3402413.3402422","relation":{},"ISSN":["0146-4833"],"issn-type":[{"value":"0146-4833","type":"print"}],"subject":[],"published":{"date-parts":[[2020,5,22]]},"assertion":[{"value":"2020-05-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}