{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:48:41Z","timestamp":1772909321369,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":24,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,7]],"date-time":"2020-11-07T00:00:00Z","timestamp":1604707200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409767","type":"proceedings-article","created":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T06:03:47Z","timestamp":1604815427000},"page":"469-480","source":"Crossref","is-referenced-by-count":25,"title":["Community expectations for research artifacts and evaluation processes"],"prefix":"10.1145","author":[{"given":"Ben","family":"Hermann","sequence":"first","affiliation":[{"name":"University of Paderborn, Germany"}]},{"given":"Stefan","family":"Winter","sequence":"additional","affiliation":[{"name":"TU Darmstadt, Germany"}]},{"given":"Janet","family":"Siegmund","sequence":"additional","affiliation":[{"name":"TU Chemnitz, Germany"}]}],"member":"320","published-online":{"date-parts":[[2020,11,7]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1515\/ci.2008.30.6.21"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799939"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340571"},{"key":"e_1_3_2_2_5_1","unstructured":"Karl Broman Mine Cetinkaya-Rundel Amy Nussbaum Christopher Paciorek Roger Peng Daniel Turek and Hadley Wickham. 2017. Recommendations to funding agencies for supporting reproducible research. https:\/\/www.amstat. org\/asa\/files\/pdfs\/POL-ReproducibleResearchRecommendations.pdf. Accessed: 2020-09-03.  Karl Broman Mine Cetinkaya-Rundel Amy Nussbaum Christopher Paciorek Roger Peng Daniel Turek and Hadley Wickham. 2017. Recommendations to funding agencies for supporting reproducible research. https:\/\/www.amstat. org\/asa\/files\/pdfs\/POL-ReproducibleResearchRecommendations.pdf. Accessed: 2020-09-03."},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2017.79"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2812803"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4016657"},{"key":"e_1_3_2_2_9_1","unstructured":"Association for Computing Machinery. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging. Accessed: 2020-09-03.  Association for Computing Machinery. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging. Accessed: 2020-09-03."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106305"},{"key":"e_1_3_2_2_11_1","unstructured":"Matthias Hauswirth. [n.d.]. Artifact Evaluation. http:\/\/evaluate.inf. usi.ch\/ artifacts. Accessed 2020-09-03.  Matthias Hauswirth. [n.d.]. Artifact Evaluation. http:\/\/evaluate.inf. usi.ch\/ artifacts. Accessed 2020-09-03."},{"key":"e_1_3_2_2_12_1","unstructured":"Ben Hermann Stefan Winter and Janet Siegmund. 2020. Community Expectations for Research Artifacts and Evaluation Processes-Data & Scripts. https:\/\/doi.org\/ 10.5281\/zenodo.3951724 10.5281\/zenodo.3951724  Ben Hermann Stefan Winter and Janet Siegmund. 2020. Community Expectations for Research Artifacts and Evaluation Processes-Data & Scripts. https:\/\/doi.org\/ 10.5281\/zenodo.3951724 10.5281\/zenodo.3951724"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Robert Heum\u00fcller Sebastian Nielebock Jacob Kr\u00fcger and Frank Ortmeier. 2020. Publish or Perish but do not Forget your Software Artifacts. Empirical Software Engineering ( 2020 ). https:\/\/doi.org\/10.1007\/s10664-020-09851-6 Preprint. 10.1007\/s10664-020-09851-6  Robert Heum\u00fcller Sebastian Nielebock Jacob Kr\u00fcger and Frank Ortmeier. 2020. Publish or Perish but do not Forget your Software Artifacts. Empirical Software Engineering ( 2020 ). https:\/\/doi.org\/10.1007\/s10664-020-09851-6 Preprint. 10.1007\/s10664-020-09851-6","DOI":"10.1007\/s10664-020-09851-6"},{"key":"e_1_3_2_2_14_1","volume-title":"The Encyclopedia of Human-Computer Interaction","author":"Hudson William"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Natalia Juristo and Sira Vegas. 2011. The Role of Non-exact Replications in Software Engineering Experiments. Empirical Software Engineering 16 3 ( 2011 ) 295-324. https:\/\/doi.org\/10.1007\/s10664-010-9141-9 10.1007\/s10664-010-9141-9  Natalia Juristo and Sira Vegas. 2011. The Role of Non-exact Replications in Software Engineering Experiments. Empirical Software Engineering 16 3 ( 2011 ) 295-324. https:\/\/doi.org\/10.1007\/s10664-010-9141-9 10.1007\/s10664-010-9141-9","DOI":"10.1007\/s10664-010-9141-9"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2658987"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368115"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"crossref","unstructured":"Daniel M\u00e9ndez Fern\u00e1ndez Wolfgang B\u00f6hm Andreas Vogelsang Jakob Mund Manfred Broy Marco Kuhrmann and Thorsten Weyer. 2019. Artefacts in software engineering: a fundamental positioning. Software & Systems Modeling 18 5 ( 2019 ) 2777-2786.  Daniel M\u00e9ndez Fern\u00e1ndez Wolfgang B\u00f6hm Andreas Vogelsang Jakob Mund Manfred Broy Marco Kuhrmann and Thorsten Weyer. 2019. Artefacts in software engineering: a fundamental positioning. Software & Systems Modeling 18 5 ( 2019 ) 2777-2786.","DOI":"10.1007\/s10270-019-00714-3"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"crossref","unstructured":"Daniel M\u00e9ndez Fern\u00e1ndez Martin Monperrus Robert Feldt and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24 3 ( 2019 ) 1057-1060. https:\/\/doi.org\/10.1007\/s10664-019-09712-x 10.1007\/s10664-019-09712-x  Daniel M\u00e9ndez Fern\u00e1ndez Martin Monperrus Robert Feldt and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24 3 ( 2019 ) 1057-1060. https:\/\/doi.org\/10.1007\/s10664-019-09712-x 10.1007\/s10664-019-09712-x","DOI":"10.1007\/s10664-019-09712-x"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463348"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"crossref","unstructured":"Forrest J Shull Jefrey C Carver Sira Vegas and Natalia Juristo. 2008. The role of replications in Empirical Software Engineering. Empirical Software Engineering 13 2 ( 2008 ) 211-218. https:\/\/doi.org\/10.1007\/s10664-008-9060-1 10.1007\/s10664-008-9060-1  Forrest J Shull Jefrey C Carver Sira Vegas and Natalia Juristo. 2008. The role of replications in Empirical Software Engineering. Empirical Software Engineering 13 2 ( 2008 ) 211-218. https:\/\/doi.org\/10.1007\/s10664-008-9060-1 10.1007\/s10664-008-9060-1","DOI":"10.1007\/s10664-008-9060-1"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2818754.2818759"},{"key":"e_1_3_2_2_23_1","volume-title":"Claire Le Goues, and Michael Hilton","author":"Timperley Christopher S.","year":"2020"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376448"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409767","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409767","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:58Z","timestamp":1750197718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,7]]},"references-count":24,"alternative-id":["10.1145\/3368089.3409767","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409767","relation":{},"subject":[],"published":{"date-parts":[[2020,11,7]]}}}