{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T02:43:51Z","timestamp":1777603431700,"version":"3.51.4"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100005304","name":"Agence Nationale de la Recherche","doi-asserted-by":"publisher","award":["ANR-16-CE23-0009"],"award-info":[{"award-number":["ANR-16-CE23-0009"]}],"id":[{"id":"10.13039\/501100005304","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2020,8,31]]},"abstract":"<jats:p>Being able to duplicate published research results is an important process of conducting research whether to build upon these findings or to compare with them. This process is called \"replicability\" when using the original authors' artifacts (e.g., code), or \"reproducibility\" otherwise (e.g., re-implementing algorithms). Reproducibility and replicability of research results have gained a lot of interest recently with assessment studies being led in various fields, and they are often seen as a trigger for better result diffusion and transparency. In this work, we assess replicability in Computer Graphics, by evaluating whether the code is available and whether it works properly. As a proxy for this field we compiled, ran and analyzed 151 codes out of 374 papers from 2014, 2016 and 2018 SIGGRAPH conferences. This analysis shows a clear increase in the number of papers with available and operational research codes with a dependency on the subfields, and indicates a correlation between code replicability and citation count. We further provide an interactive tool to explore our results and evaluation data.<\/jats:p>","DOI":"10.1145\/3386569.3392413","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T11:44:27Z","timestamp":1597232667000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Code replicability in computer graphics"],"prefix":"10.1145","volume":"39","author":[{"given":"Nicolas","family":"Bonneel","sequence":"first","affiliation":[{"name":"Univ Lyon, CNRS, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Coeurjolly","sequence":"additional","affiliation":[{"name":"Univ Lyon, CNRS, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julie","family":"Digne","sequence":"additional","affiliation":[{"name":"Univ Lyon, CNRS, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicolas","family":"Mellado","sequence":"additional","affiliation":[{"name":"Univ Toulouse, CNRS, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging (accessed","author":"ACM.","year":"2019","unstructured":"ACM. 2016. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging (accessed Dec. 2019)."},{"key":"e_1_2_2_2_1","volume-title":"Open Access to ACM SIGGRAPH-Sponsored Content. https:\/\/www.siggraph.org\/learn\/conference-content\/ (accessed","author":"ACM.","year":"2020","unstructured":"ACM. 2020. Open Access to ACM SIGGRAPH-Sponsored Content. https:\/\/www.siggraph.org\/learn\/conference-content\/ (accessed Apr. 2020)."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1038\/541269a"},{"key":"e_1_2_2_5_1","volume-title":"Terminologies for reproducible research. arXiv preprint arXiv:1802.03311","author":"Barba Lorena A","year":"2018","unstructured":"Lorena A Barba. 2018. Terminologies for reproducible research. arXiv preprint arXiv:1802.03311 (2018)."},{"key":"e_1_2_2_6_1","volume-title":"Reproducibility: Six red flags for suspect work. Nature 497, 7450","author":"Begley C Glenn","year":"2013","unstructured":"C Glenn Begley. 2013. Reproducibility: Six red flags for suspect work. Nature 497, 7450 (2013), 433."},{"key":"e_1_2_2_7_1","volume-title":"Drug development: Raise standards for preclinical cancer research. Nature 483, 7391","author":"Glenn Begley C","year":"2012","unstructured":"C Glenn Begley and Lee M Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature 483, 7391 (2012), 531."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-017-0189-z"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-018-0399-z"},{"key":"e_1_2_2_11_1","volume-title":"Open Science Framework. https:\/\/www.osf.io (accessed","author":"Center for Open Science. 2015.","year":"2019","unstructured":"Center for Open Science. 2015. Open Science Framework. https:\/\/www.osf.io (accessed Dec. 2019)."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41567-018-0342-2"},{"key":"e_1_2_2_13_1","volume-title":"SEG Technical Program Expanded Abstracts","author":"Claerbout Jon F","year":"1992","unstructured":"Jon F Claerbout and Martin Karrenbach. 1992. Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, 601--604."},{"key":"e_1_2_2_14_1","volume-title":"What if there were no significance tests?","author":"Cohen Jacob","unstructured":"Jacob Cohen. 2016. The earth is round (p&lt;. 05). In What if there were no significance tests? Routledge, 69--82."},{"key":"e_1_2_2_15_1","volume-title":"An Overview of Platforms for Reproducible Research and Augmented Publications. In International Workshop on Reproducible Research in Pattern Recognition. Springer, 25--39","author":"Colom Miguel","year":"2018","unstructured":"Miguel Colom, Bertrand Kerautret, and Adrien Kr\u00e4henb\u00fchl. 2018. An Overview of Platforms for Reproducible Research and Augmented Publications. In International Workshop on Reproducible Research in Pattern Recognition. Springer, 25--39."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/NTMS.2015.7266500"},{"key":"e_1_2_2_17_1","volume-title":"Software Heritage: Why and How to Preserve Software Source Code. In iPRES 2017: 14th International Conference on Digital Preservation (2017-09-25). Kyoto, Japan.","author":"Cosmo Roberto Di","year":"2017","unstructured":"Roberto Di Cosmo and Stefano Zacchiroli. 2017. Software Heritage: Why and How to Preserve Software Source Code. In iPRES 2017: 14th International Conference on Digital Preservation (2017-09-25). Kyoto, Japan."},{"key":"e_1_2_2_18_1","volume-title":"The new watchdogs' vision of science: A roundtable with Ivan Oransky (Retraction Watch) and Brandon Stell (PubPeer). Social studies of science 48, 1","author":"Didier Emmanuel","year":"2018","unstructured":"Emmanuel Didier and Catherine Guaspare-Cartron. 2018. The new watchdogs' vision of science: A roundtable with Ivan Oransky (Retraction Watch) and Brandon Stell (PubPeer). Social studies of science 48, 1 (2018), 165--167."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0172650"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1602413113"},{"key":"e_1_2_2_21_1","volume-title":"Call for replication repository. https:\/\/github.com\/ReScience\/call-for-replication (accessed","year":"2019","unstructured":"Github. 2016. Call for replication repository. https:\/\/github.com\/ReScience\/call-for-replication (accessed Dec. 2019)."},{"key":"e_1_2_2_22_1","volume-title":"Github Archive Program. https:\/\/archiveprogram.github.com\/ (accessed","year":"2020","unstructured":"Github. 2020. Github Archive Program. https:\/\/archiveprogram.github.com\/ (accessed Jan. 2020)."},{"key":"e_1_2_2_23_1","volume-title":"What does research reproducibility mean? Science translational medicine 8, 341","author":"Goodman Steven N","year":"2016","unstructured":"Steven N Goodman, Daniele Fanelli, and John PA Ioannidis. 2016. What does research reproducibility mean? Science translational medicine 8, 341 (2016)."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11503"},{"key":"e_1_2_2_25_1","doi-asserted-by":"crossref","unstructured":"Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis.","DOI":"10.1126\/science.359.6377.725"},{"key":"e_1_2_2_26_1","volume-title":"Reproducible Research in Pattern Recognition","author":"Kerautret Bertrand","unstructured":"Bertrand Kerautret, Miguel Colom, Daniel Lopresti, Pascal Monasse, and Hugues Talbot. 2019. Reproducible Research in Pattern Recognition. Springer."},{"key":"e_1_2_2_27_1","volume-title":"Reproducible Research in Pattern Recognition","author":"Kerautret Bertrand","unstructured":"Bertrand Kerautret, Miguel Colom, and Pascal Monasse. 2017. Reproducible Research in Pattern Recognition. Springer."},{"key":"e_1_2_2_28_1","volume-title":"Sinan Alper, Mark Aveyard, Jordan R Axt, Mayowa T Babalola, \u0160t\u011bp\u00e1n Bahn\u00edk, et al.","author":"Klein Richard A","year":"2018","unstructured":"Richard A Klein, Michelangelo Vianello, Fred Hasselman, Byron G Adams, Reginald B Adams Jr, Sinan Alper, Mark Aveyard, Jordan R Axt, Mayowa T Babalola, \u0160t\u011bp\u00e1n Bahn\u00edk, et al. 2018. Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychol. Sci. 1, 4 (2018)."},{"key":"e_1_2_2_29_1","volume-title":"About Artifact Evaluation. https:\/\/www.artifact-eval.org\/about.html (accessed","author":"Krishnamurthi Shriram","year":"2020","unstructured":"Shriram Krishnamurthi. 2020. About Artifact Evaluation. https:\/\/www.artifact-eval.org\/about.html (accessed Apr. 2020)."},{"key":"e_1_2_2_30_1","volume-title":"Managing Microsoft's Conference Management Toolkit with Python for NIPS","author":"Lawrence Neil D.","year":"2014","unstructured":"Neil D. Lawrence. 2014. Managing Microsoft's Conference Management Toolkit with Python for NIPS 2014. https:\/\/github.com\/sods\/conference (accessed Apr. 2020)."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612460688"},{"key":"e_1_2_2_32_1","volume-title":"Abandon statistical significance. The American Statistician 73, sup1","author":"McShane Blakeley B","year":"2019","unstructured":"Blakeley B McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L Tackett. 2019. Abandon statistical significance. The American Statistician 73, sup1 (2019), 235--245."},{"key":"e_1_2_2_33_1","unstructured":"Engineering National Academies of Sciences Medicine et al. 2019. Reproducibility and replicability in science. National Academies Press."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-06012-8"},{"key":"e_1_2_2_35_1","volume-title":"Likert scales, levels of measurement and the \"laws\" of statistics. Advances in health sciences education 15, 5","author":"Norman Geoff","year":"2010","unstructured":"Geoff Norman. 2010. Likert scales, levels of measurement and the \"laws\" of statistics. Advances in health sciences education 15, 5 (2010), 625--632."},{"key":"e_1_2_2_36_1","doi-asserted-by":"crossref","unstructured":"Open Science Collaboration et al. 2015. Estimating the reproducibility of psychological science. Science 349 6251 (2015) aac4716.","DOI":"10.1126\/science.aac4716"},{"key":"e_1_2_2_37_1","unstructured":"Daniele Panozzo. 2016. Graphics replicability stamp initiative. http:\/\/www.replicabilitystamp.org\/ (accessed Dec. 2019)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612463401"},{"key":"e_1_2_2_39_1","volume-title":"Rosemary Nan Ke, and Hugo Larochelle","author":"Pineau Joelle","year":"2019","unstructured":"Joelle Pineau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. 2019. ICLR Reproducibility Challenge 2019. ReScience C (2019)."},{"key":"e_1_2_2_40_1","volume-title":"Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in neuroinformatics 11","author":"Plesser Hans E","year":"2018","unstructured":"Hans E Plesser. 2018. Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in neuroinformatics 11 (2018), 76."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/369545.369555"},{"key":"e_1_2_2_42_1","volume-title":"Recommended Data Repositories. https:\/\/www.nature.com\/sdata\/policies\/repositories (accessed","author":"Nature Scientific Data","year":"2019","unstructured":"Scientific Data (Nature Research). 2014. Recommended Data Repositories. https:\/\/www.nature.com\/sdata\/policies\/repositories (accessed Dec. 2019)."},{"key":"e_1_2_2_43_1","volume-title":"ICML 2013 Peer Review Workshop (2013","author":"Soergel David","year":"2013","unstructured":"David Soergel, Adam Saunders, and Andrew McCallum. 2013. Open Scholarship and Peer Review: a Time for Experimentation. ICML 2013 Peer Review Workshop (2013). https:\/\/openreview.net\/ (accessed Dec. 2019)."},{"key":"e_1_2_2_44_1","volume-title":"Assessing data availability and research reproducibility in hydrology and water resources. Scientific data 6","author":"Stagge James H","year":"2019","unstructured":"James H Stagge, David E Rosenberg, Adel M Abdallah, Hadia Akbar, Nour A Attallah, and Ryan James. 2019. Assessing data availability and research reproducibility in hydrology and water resources. Scientific data 6 (2019)."},{"key":"e_1_2_2_45_1","volume-title":"Open Peer Review. https:\/\/royalsocietypublishing.org\/rspa\/open-peer-review (accessed","author":"The Royal Society Publishing. 2020.","year":"2020","unstructured":"The Royal Society Publishing. 2020. Open Peer Review. https:\/\/royalsocietypublishing.org\/rspa\/open-peer-review (accessed Jan. 2020)."},{"key":"e_1_2_2_46_1","volume-title":"Proceedings of the 7th International Web Archiving Workshop. 27--37","author":"Tofel Brad","year":"2007","unstructured":"Brad Tofel. 2007. Wayback' for accessing web archives. In Proceedings of the 7th International Web Archiving Workshop. 27--37."},{"key":"e_1_2_2_47_1","unstructured":"Patrick Vandewalle. 2019. Code availability for image processing papers: a status update. https:\/\/lirias.kuleuven.be\/retrieve\/541895$$DVandewalle_onlinecode_TIP_SITB19.pdf"},{"key":"e_1_2_2_48_1","volume-title":"Gene name errors are widespread in the scientific literature. Genome biology 17, 1","author":"Ziemann Mark","year":"2016","unstructured":"Mark Ziemann, Yotam Eren, and Assam El-Osta. 2016. Gene name errors are widespread in the scientific literature. Genome biology 17, 1 (2016), 177."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392413","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3386569.3392413","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T05:34:31Z","timestamp":1750829671000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392413"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,12]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,8,31]]}},"alternative-id":["10.1145\/3386569.3392413"],"URL":"https:\/\/doi.org\/10.1145\/3386569.3392413","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,12]]},"assertion":[{"value":"2020-08-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}