{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T04:31:44Z","timestamp":1774153904927,"version":"3.50.1"},"reference-count":47,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Data Science"],"published-print":{"date-parts":[[2021,5,21]]},"abstract":"<jats:p>Adopting open science principles can be challenging, requiring conceptual education and training in the use of new tools. This paper introduces the Workflow for Open Reproducible Code in Science (WORCS): A step-by-step procedure that researchers can follow to make a research project open and reproducible. This workflow intends to lower the threshold for adoption of open science principles. It is based on established best practices, and can be used either in parallel to, or in absence of, top-down requirements by journals, institutions, and funding bodies. To facilitate widespread adoption, the WORCS principles have been implemented in the R package worcs, which offers an RStudio project template and utility functions for specific workflow steps. This paper introduces the conceptual workflow, discusses how it meets different standards for open science, and addresses the functionality provided by the R implementation, worcs. This paper is primarily targeted towards scholars conducting research projects in R, conducting research that involves academic prose, analysis code, and tabular data. However, the workflow is flexible enough to accommodate other scenarios, and offers a starting point for customized solutions. The source code for the R package and manuscript, and a list of examplesof WORCS projects , are available at https:\/\/github.com\/cjvanlissa\/worcs .<\/jats:p>","DOI":"10.3233\/ds-210031","type":"journal-article","created":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T12:06:47Z","timestamp":1613736407000},"page":"29-49","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":30,"title":["WORCS: A workflow for open reproducible code in science"],"prefix":"10.1177","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0808-5024","authenticated-orcid":false,"given":"Caspar J.","family":"Van Lissa","sequence":"first","affiliation":[{"name":"Department of Methodology & Statistics, Utrecht University, The Netherlands and Open Science Community Utrecht, The Netherlands. E-mail:\u00a0"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8765-6982","authenticated-orcid":false,"given":"Andreas M.","family":"Brandmaier","sequence":"additional","affiliation":[{"name":"Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany and London, UK."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3997-1173","authenticated-orcid":false,"given":"Loek","family":"Brinkman","sequence":"additional","affiliation":[{"name":"Open Science Community Utrecht, The Netherlands and University Medical Center, Utrecht University, The Netherlands."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1953-5606","authenticated-orcid":false,"given":"Anna-Lena","family":"Lamprecht","sequence":"additional","affiliation":[{"name":"Open Science Community Utrecht, The Netherlands and Department of Information and Computing Sciences, Utrecht University, The Netherlands."}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7813-818X","authenticated-orcid":false,"given":"Aaron","family":"Peikert","sequence":"additional","affiliation":[{"name":"Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1166-1424","authenticated-orcid":false,"given":"Marijn E.","family":"Struiksma","sequence":"additional","affiliation":[{"name":"Open Science Community Utrecht, The Netherlands and Utrecht Institute of Linguistics OTS, Utrecht University, The Netherlands."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5023-4601","authenticated-orcid":false,"given":"Barbara M.I.","family":"Vreede","sequence":"additional","affiliation":[{"name":"University Library, Utrecht University, The Netherlands and Netherlands eScience Center, The\u00a0Netherlands."}]}],"member":"179","published-online":{"date-parts":[[2021,2,19]]},"reference":[{"key":"ref001","doi-asserted-by":"publisher","DOI":"10.31219\/osf.io\/sm78t"},{"key":"ref002","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-019-0772-6"},{"key":"ref003","doi-asserted-by":"publisher","DOI":"10.1080\/1047840X.2012.705133"},{"key":"ref004","unstructured":"J.\u00a0Allaire, Y.\u00a0Xie, R Foundation, H.\u00a0Wickham, Journal of Statistical SoftwareR.\u00a0Vaidyanathanet al.Rticles: Article formats for r markdown 2020, Retrieved from https:\/\/CRAN.R-project.org\/package=rticles."},{"key":"ref005","unstructured":"J.J.\u00a0Allaire, K.\u00a0Ushey, RStudio and Y.\u00a0Tang, R Markdown Python engine, 2020, Retrieved January 13, 2020, from https:\/\/rstudio.github.io\/reticulate\/articles\/r_markdown.html."},{"key":"ref006","unstructured":"F.\u00a0Aust, Prereg: R Markdown templates to preregister scientific studies (version 0.4.0), 2019, Retrieved from https:\/\/CRAN.R-project.org\/package=prereg."},{"key":"ref007","unstructured":"F.\u00a0Aust and M.\u00a0Barth, Papaja: Prepare reproducible APA journal articles with R Markdown, 2020, (Version 0.1.0.9842), (Original work published 2014), Retrieved from https:\/\/github.com\/crsh\/papaja."},{"key":"ref008","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.1212496"},{"key":"ref009","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1004668"},{"key":"ref010","unstructured":"C.T.\u00a0Brown, How I learned to stop worrying and love the coming archivability crisis in scientific software, 2017, Retrieved January 13, 2020, from http:\/\/ivory.idyll.org\/blog\/2017-pof-software-archivability.html."},{"key":"ref011","doi-asserted-by":"publisher","DOI":"10.1186\/s40359-016-0134-3"},{"key":"ref012","doi-asserted-by":"publisher","DOI":"10.1511\/2014.111.460"},{"key":"ref013","unstructured":"K.\u00a0Goldfeld, Simstudy: Simulation of study data, 2020, Retrieved from https:\/\/CRAN.R-project.org\/package=simstudy."},{"key":"ref014","unstructured":"G.\u00a0Grolemund and H.\u00a0Wickham, R for Data Science, O\u2019Reilly, 2017, Retrieved from https:\/\/r4ds.had.co.nz\/."},{"key":"ref015","unstructured":"M.\u00a0Hallquist, J.\u00a0Wiley and C.J.\u00a0Van Lissa, MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus (Version 0.7-3), 2018, Retrieved from https:\/\/CRAN.R-project.org\/package=MplusAutomation."},{"key":"ref016","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430953"},{"key":"ref017","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr0203_4"},{"key":"ref018","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002456"},{"key":"ref019","doi-asserted-by":"publisher","DOI":"10.3233\/DS-190026"},{"key":"ref020","unstructured":"W.J.M.\u00a0Levelt, E.\u00a0Noort and P.J.D.\u00a0Drenth, Failing science: The fraudulent research practices of social psychologist Diederik Stapel (Falende wetenschap: De frauduleuze onderzoekspraktijken van sociaal-psycholoog Diederik Stapel), 2012, Retrieved from https:\/\/www.onderwijsbrabant.nl\/sites\/default\/files\/eindrapport_stapel_nov_2012.pdf."},{"key":"ref021","unstructured":"D.S.\u00a0Lindsay, D.J.\u00a0Simons and S.O.\u00a0Lilienfeld, Research preregistration 101, APS Observer29(10) (2016), Retrieved from https:\/\/www.psychologicalscience.org\/observer\/research-preregistration-101."},{"key":"ref022","unstructured":"R.A.\u00a0Muenchen, The popularity of data science software, 2012, April 25, Retrieved January 8, 2020, from http:\/\/r4stats.com\/articles\/popularity\/."},{"key":"ref023","doi-asserted-by":"publisher","DOI":"10.1126\/science.aab2374"},{"key":"ref024","doi-asserted-by":"publisher","DOI":"10.1080\/1047840X.2012.692215"},{"key":"ref025","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v074.i11"},{"key":"ref026","doi-asserted-by":"crossref","unstructured":"J.\u00a0Ooms, Gert: Simple Git client for r, 2019, Retrieved from https:\/\/CRAN.R-project.org\/package=gert.","DOI":"10.32614\/CRAN.package.gert"},{"key":"ref027","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-019-0629-z"},{"key":"ref028","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/8xzqy"},{"key":"ref029","unstructured":"A.\u00a0Peikert, A.M.\u00a0Brandmaier and C.J.\u00a0Van Lissa, Repro: Automated setup of reproducible workflows and their dependencies, 2020, Retrieved from https:\/\/github.com\/aaronpeikert\/repro."},{"key":"ref030","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.3696"},{"key":"ref031","unstructured":"R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2020. Retrieved from https:\/\/www.R-project.org\/."},{"key":"ref032","doi-asserted-by":"publisher","DOI":"10.1186\/1751-0473-8-7"},{"key":"ref033","doi-asserted-by":"crossref","unstructured":"R.\u00a0Rivest, The MD5 message-digest algorithm, MIT Laboratory for Computer Science and RSA Data Security, Inc., 1992. Available at: https:\/\/doi.org\/10.17487\/RFC1321.","DOI":"10.17487\/rfc1321"},{"key":"ref034","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00978"},{"key":"ref035","unstructured":"RStudio Team, RStudio: Integrated Development Environment for R, RStudio, Inc., Boston, MA, 2015. Retrieved from http:\/\/www.rstudio.com\/."},{"key":"ref036","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-122216-011845"},{"key":"ref037","unstructured":"J.\u00a0Sondervan, J.\u00a0Bosman, B.\u00a0Kramer, L.\u00a0Brinkman, M.\u00a0Imming and A.\u00a0Versteeg, The COVID-19 pandemic stresses the societal importance of open science,\n                      ScienceGuide\n                      , 2020, April 3, Retrieved from https:\/\/www.scienceguide.nl\/2020\/04\/dire-times-of-covid-19-stress-the-societal-importance-of-open-science\/."},{"key":"ref038","doi-asserted-by":"publisher","DOI":"10.1126\/science.aah6168"},{"key":"ref039","unstructured":"J.\u00a0Tennant, Open science is just good science, 2018, TU Delft, Retrieved from https:\/\/figshare.com\/articles\/Open_Science_is_just_good_science_pptx\/5783004."},{"key":"ref040","doi-asserted-by":"crossref","unstructured":"K.\u00a0Ushey, 2020, Renv: Project environments (Version 0.12.0). Retrieved from https:\/\/CRAN.R-project.org\/package=renv.","DOI":"10.32614\/CRAN.package.renv"},{"key":"ref041","doi-asserted-by":"crossref","unstructured":"C.J.\u00a0Van Lissa, A.\u00a0Peikert and A.M.\u00a0Brandmaier, Worcs: Workflow for open reproducible code in science (Version 0.1.5), 2020, Retrieved from https:\/\/cran.r-project.org\/web\/packages\/worcs\/index.html.","DOI":"10.32614\/CRAN.package.worcs"},{"key":"ref042","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2016.03.004"},{"key":"ref043","doi-asserted-by":"publisher","DOI":"10.1177\/2515245919848684"},{"key":"ref044","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v059.i10"},{"key":"ref045","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18"},{"key":"ref046","unstructured":"M.N.\u00a0Wright and A.\u00a0Ziegler, Ranger: A fast implementation of random forests for high dimensional data in C++ and R, 2015, August 18, Retrieved from http:\/\/arxiv.org\/abs\/1508.04409."},{"key":"ref047","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Xie, J.J.\u00a0Allaire and G.\u00a0Grolemund, R Markdown: The Definitive Guide, Chapman and Hall\/CRC, 2018. Retrieved from https:\/\/bookdown.org\/yihui\/rmarkdown\/.","DOI":"10.1201\/9781138359444"}],"container-title":["Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/DS-210031","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/DS-210031","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/DS-210031","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,10]],"date-time":"2025-03-10T19:14:20Z","timestamp":1741634060000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/DS-210031"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,19]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,5,21]]}},"alternative-id":["10.3233\/DS-210031"],"URL":"https:\/\/doi.org\/10.3233\/ds-210031","relation":{"has-preprint":[{"id-type":"doi","id":"10.31234\/osf.io\/k4wde","asserted-by":"object"}]},"ISSN":["2451-8484","2451-8492"],"issn-type":[{"value":"2451-8484","type":"print"},{"value":"2451-8492","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,19]]}}}