{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T02:43:34Z","timestamp":1793155414322,"version":"4.5.0"},"reference-count":55,"publisher":"Public Library of Science (PLoS)","issue":"3","license":[{"start":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T00:00:00Z","timestamp":1616025600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>A systematic and reproducible \u201cworkflow\u201d\u2014the process that moves a scientific investigation from raw data to coherent research question to insightful contribution\u2014should be a fundamental part of academic data-intensive research practice. In this paper, we elaborate basic principles of a reproducible data analysis workflow by defining 3 phases: the Explore, Refine, and Produce Phases. Each phase is roughly centered around the audience to whom research decisions, methodologies, and results are being immediately communicated. Importantly, each phase can also give rise to a number of research products beyond traditional academic publications. Where relevant, we draw analogies between design principles and established practice in software development. The guidance provided here is not intended to be a strict rulebook; rather, the suggestions for practices and tools to advance reproducible, sound data-intensive analysis may furnish support for both students new to research and current researchers who are new to data-intensive work.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1008770","type":"journal-article","created":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T17:30:02Z","timestamp":1616088602000},"page":"e1008770","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":40,"title":["Principles for data analysis workflows"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1693-8058","authenticated-orcid":true,"given":"Sara","family":"Stoudt","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2108-6459","authenticated-orcid":true,"given":"V\u00e1leri N.","family":"V\u00e1squez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4296-998X","authenticated-orcid":true,"given":"Ciera C.","family":"Martinez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,3,18]]},"reference":[{"key":"pcbi.1008770.ref001","doi-asserted-by":"crossref","first-page":"1266","DOI":"10.1186\/s40064-016-2888-8","article-title":"A preliminary review of influential works in data-driven discovery","volume":"5","author":"M Stalzer","year":"2016","journal-title":"Springerplus"},{"key":"pcbi.1008770.ref002","doi-asserted-by":"crossref","DOI":"10.1093\/gigascience\/giz054","article-title":"Software engineering for scientific big data analysis","volume":"8","author":"BA Gr\u00fcning","year":"2019","journal-title":"Gigascience"},{"key":"pcbi.1008770.ref003","unstructured":"Robinson E, Nolis J. Build a Career in Data Science. Simon and Schuster; 2020."},{"key":"pcbi.1008770.ref004","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1186\/s13059-020-02133-w","article-title":"A hypothesis is a liability","volume":"21","author":"I Yanai","year":"2020","journal-title":"Genome Biol"},{"key":"pcbi.1008770.ref005","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1162\/074793601750357196","article-title":"Designerly Ways of Knowing: Design Discipline Versus Design Science","volume":"17","author":"N Cross","year":"2001","journal-title":"Design Issues"},{"key":"pcbi.1008770.ref006","unstructured":"Terence S. An Extensive Step by Step Guide to Exploratory Data Analysis. 2020 [cited 2020 Jun 15]. https:\/\/towardsdatascience.com\/an-extensive-guide-to-exploratory-data-analysis-ddd99a03199e."},{"key":"pcbi.1008770.ref007","doi-asserted-by":"crossref","first-page":"341ps12","DOI":"10.1126\/scitranslmed.aaf5027","article-title":"What does research reproducibility mean?","volume":"8","author":"SN Goodman","year":"2016","journal-title":"Sci Transl Med"},{"key":"pcbi.1008770.ref008","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1145\/356635.356640","article-title":"Structured Programming with go to Statements","volume":"6","author":"DE Knuth","year":"1974","journal-title":"ACM Comput Surv"},{"key":"pcbi.1008770.ref009","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MS.2009.147","article-title":"Writing Code to Prototype, Ideate, and Discover","volume":"26","author":"J Brandt","year":"2009","journal-title":"IEEE Softw"},{"key":"pcbi.1008770.ref010","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/MPRV.2008.54","article-title":"Hacking, Mashing, Gluing: Understanding Opportunistic Design","volume":"7","author":"B Hartmann","year":"2008","journal-title":"IEEE Pervasive Comput"},{"key":"pcbi.1008770.ref011","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/d41586-018-07196-1","article-title":"Why Jupyter is data scientists\u2019 computational notebook of choice","volume":"563","author":"JM Perkel","year":"2018","journal-title":"Nature"},{"key":"pcbi.1008770.ref012","doi-asserted-by":"crossref","first-page":"3920","DOI":"10.1073\/pnas.1901326117","article-title":"Veridical data science","volume":"117","author":"B Yu","year":"2020","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1008770.ref013","unstructured":"Bostock MA. Better Way to Code\u2014Mike Bostock\u2014Medium. 2017 [cited 2020 Jun 15]. https:\/\/medium.com\/@mbostock\/a-better-way-to-code-2b1d2876a3a0."},{"key":"pcbi.1008770.ref014","unstructured":"van der Plas F. Pluto.jl. Github. https:\/\/github.com\/fonsp\/Pluto.jl."},{"key":"pcbi.1008770.ref015","unstructured":"Best Practices for Writing R Code\u2013Programming with R. [cited 15 Jun 2020]. https:\/\/swcarpentry.github.io\/r-novice-inflammation\/06-best-practices-R\/"},{"key":"pcbi.1008770.ref016","unstructured":"PyCon 2019. Jes Ford\u2014Getting Started Testing in Data Science\u2014PyCon 2019. Youtube; 5 May 2019 [cited 2020 Feb 20]. https:\/\/www.youtube.com\/watch?v=0ysyWk-ox-8"},{"key":"pcbi.1008770.ref017","doi-asserted-by":"crossref","unstructured":"Hook D, Kelly D. Testing for trustworthiness in scientific software. 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering. 2009. pp. 59\u201364.","DOI":"10.1109\/SECSE.2009.5069163"},{"key":"pcbi.1008770.ref018","unstructured":"Oh J-H. Check Yo\u2019 Data Before You Wreck Yo\u2019 Results. In: Medium [Internet]. ACLU Tech & Analytics; 24 Jan 2020 [cited 2020 Apr 9]. https:\/\/medium.com\/aclu-tech-analytics\/check-yo-data-before-you-wreck-yo-results-53f0e919d0b9."},{"key":"pcbi.1008770.ref019","unstructured":"Gelfand S. comparing two data frames: one #rstats, many ways! | Sharla Gelfand. In: Sharla Gelfand [Internet]. Sharla Gelfand; 17 Feb 2020 [cited 2020 Apr 20]. https:\/\/sharla.party\/post\/comparing-two-dfs\/."},{"key":"pcbi.1008770.ref020","unstructured":"Gelfand S. Don\u2019t repeat yourself, talk to yourself! Repeated reporting in the R universe | Sharla Gelfand. In: Sharla Gelfand [Internet]. 30 Jan 2020 [cited 2020 Apr 20]. https:\/\/sharla.party\/talk\/2020-01-01-rstudio-conf\/."},{"key":"pcbi.1008770.ref021","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1038\/nbt.3820","article-title":"Nextflow enables reproducible computational workflows","volume":"35","author":"PD Tommaso","year":"2017","journal-title":"Nature Biotechnol"},{"key":"pcbi.1008770.ref022","article-title":"Streamlining Data-Intensive Biology With Workflow Systems","author":"T Reiter","year":"2020","journal-title":"Cold Spring Harb Laboratory"},{"key":"pcbi.1008770.ref023","doi-asserted-by":"crossref","first-page":"2520","DOI":"10.1093\/bioinformatics\/bts480","article-title":"Snakemake\u2014a scalable bioinformatics workflow engine","volume":"28","author":"J K\u00f6ster","year":"2012","journal-title":"Bioinformatics"},{"key":"pcbi.1008770.ref024","doi-asserted-by":"crossref","first-page":"e1001745","DOI":"10.1371\/journal.pbio.1001745","article-title":"Best practices for scientific computing","volume":"12","author":"G Wilson","year":"2014","journal-title":"PLoS Biol"},{"key":"pcbi.1008770.ref025","doi-asserted-by":"crossref","first-page":"e1005510","DOI":"10.1371\/journal.pcbi.1005510","article-title":"Good enough practices in scientific computing","volume":"13","author":"G Wilson","year":"2017","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1008770.ref026","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1126\/science.1213847","article-title":"Reproducible research in computational science","volume":"334","author":"RD Peng","year":"2011","journal-title":"Science"},{"key":"pcbi.1008770.ref027","doi-asserted-by":"crossref","unstructured":"Geiger RS, Sholler D, Culich A, Martinez C, Hoces de la Guardia F, Lanusse F, et al. Challenges of Doing Data-Intensive Research in Teams, Labs, and Groups: Report from the BIDS Best Practices in Data Science Series. 2018.","DOI":"10.31235\/osf.io\/a7b3m"},{"key":"pcbi.1008770.ref028","first-page":"108","article-title":"Project Jupyter: Computational narratives as the engine of collaborative data science","volume":"11","author":"F Perez","year":"2015","journal-title":"Retrieved September"},{"key":"pcbi.1008770.ref029","doi-asserted-by":"crossref","unstructured":"Xie Y. Dynamic Documents with R and knitr. Chapman and Hall\/CRC; 2017.","DOI":"10.1201\/b15166"},{"key":"pcbi.1008770.ref030","doi-asserted-by":"crossref","first-page":"e1004385","DOI":"10.1371\/journal.pcbi.1004385","article-title":"Ten Simple Rules for a Computational Biologist\u2019s Laboratory Notebook","author":"S Schnell","year":"2015","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1008770.ref031","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/ng.295","article-title":"Repeatability of published microarray gene expression analyses","volume":"41","author":"JPA Ioannidis","year":"2009","journal-title":"Nat Genet"},{"key":"pcbi.1008770.ref032","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1038\/nrg3305","article-title":"Next-generation sequencing data interpretation: enhancing reproducibility and accessibility","volume":"13","author":"A Nekrutenko","year":"2012","journal-title":"Nat Rev Genet"},{"key":"pcbi.1008770.ref033","unstructured":"Wickham H. R Packages: Organize, Test, Document, and Share Your Code. \u201cO\u2019Reilly Media, Inc.\u201d; 2015."},{"key":"pcbi.1008770.ref034","unstructured":"Abrahamsson P, Salo O, Ronkainen J, Warsta J. Agile Software Development Methods: Review and Analysis. arXiv [cs.SE]. 2017. http:\/\/arxiv.org\/abs\/1709.08439."},{"key":"pcbi.1008770.ref035","unstructured":"Beck K, Beedle M, Van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al. Manifesto for agile software development. 2001. https:\/\/moodle2019-20.ua.es\/moodle\/pluginfile.php\/2213\/mod_resource\/content\/2\/agile-manifesto.pdf."},{"key":"pcbi.1008770.ref036","first-page":"20192047","article-title":"Games academics play and their consequences: how authorship, h-index and journal impact factors are shaping the future of academia","volume":"286","author":"CA Chapman","year":"2019","journal-title":"Proc Biol Sci"},{"key":"pcbi.1008770.ref037","article-title":"Publishing computational research\u2013A review of infrastructures for reproducible and transparent scholarly communication","author":"M Konkol","year":"2020","journal-title":"oral"},{"key":"pcbi.1008770.ref038","doi-asserted-by":"crossref","unstructured":"Sholler D, Das D, Hoces de la Guardia F, Hoffman C, Lanusse F, Varoquaux N, et al. Best Practices for Managing Turnover in Data Science Groups, Teams, and Labs. 2019.","DOI":"10.31235\/osf.io\/wsxru"},{"key":"pcbi.1008770.ref039","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1111\/acem.12482","article-title":"Using publication metrics to highlight academic productivity and research impact","volume":"21","author":"CR Carpenter","year":"2014","journal-title":"Acad Emerg Med"},{"key":"pcbi.1008770.ref040","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1093\/biostatistics\/kxq028","article-title":"An invitation to reproducible computational research","volume":"11","author":"DL Donoho","year":"2010","journal-title":"Biostatistics"},{"key":"pcbi.1008770.ref041","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1198\/106186007X178663","article-title":"Statistical Analyses and Reproducible Research","volume":"16","author":"R Gentleman","year":"2007","journal-title":"J Comput Graph Stat"},{"key":"pcbi.1008770.ref042","article-title":"Opening the Publication Process with Executable Research Compendia","author":"D N\u00fcst","year":"2017","journal-title":"D-Lib Magazine"},{"key":"pcbi.1008770.ref043","doi-asserted-by":"crossref","first-page":"2584","DOI":"10.1073\/pnas.1708290115","article-title":"An empirical analysis of journal policy effectiveness for computational reproducibility","volume":"115","author":"V Stodden","year":"2018","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1008770.ref044","unstructured":"Martin RC. Clean Code: A Handbook of Agile Software Craftsmanship. Pearson Education; 2009."},{"key":"pcbi.1008770.ref045","unstructured":"Fowler M. Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional; 2018."},{"key":"pcbi.1008770.ref046","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1126\/science.aah6168","article-title":"Enhancing reproducibility for computational methods","volume":"354","author":"V Stodden","year":"2016","journal-title":"Science"},{"key":"pcbi.1008770.ref047","doi-asserted-by":"crossref","unstructured":"Geiger RS, Cabasse C, Cullens CY, Nor\u00e9n L, Fiore-Gartland B, Das D, et al. Career Paths and Prospects in Academic Data Science: Report of the Moore-Sloan Data Science Environments Survey. 2018.","DOI":"10.31235\/osf.io\/xe823"},{"key":"pcbi.1008770.ref048","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1002\/spe.491","article-title":"Lessons learned from automating tests for an operations support system","volume":"32","author":"MA Fecko","year":"2002","journal-title":"Softw Pract Exp"},{"key":"pcbi.1008770.ref049","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1201\/9781315204970-11","volume-title":"The Craft of Model-Based Testing","author":"PC Jorgensen","year":"2017","edition":"1"},{"key":"pcbi.1008770.ref050","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/64.585103","article-title":"Functional reasoning in design","volume":"12","author":"Y Umeda","year":"1997","journal-title":"IEEE Expert"},{"key":"pcbi.1008770.ref051","unstructured":"Wikipedia contributors. Functional design. In: Wikipedia, The Free Encyclopedia [Internet]. 4 Feb 2020 [cited 21 Feb 2020]. https:\/\/en.wikipedia.org\/w\/index.php?title=Functional_design&oldid=939128138"},{"key":"pcbi.1008770.ref052","unstructured":"7 Essential Guidelines For Functional Design\u2014Smashing Magazine. In: Smashing Magazine [Internet]. 5 Aug 2008 [cited 21 Feb 2020]. https:\/\/www.smashingmagazine.com\/2008\/08\/7-essential-guidelines-for-functional-design\/"},{"key":"pcbi.1008770.ref053","doi-asserted-by":"crossref","unstructured":"Claerbout JF, Karrenbach M. Electronic documents give reproducible research a new meaning. SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists; 1992. pp. 601\u2013604.","DOI":"10.1190\/1.1822162"},{"key":"pcbi.1008770.ref054","doi-asserted-by":"crossref","unstructured":"Heroux MA, Barba L, Parashar M, Stodden V, Taufer M. Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States); 2018. https:\/\/www.osti.gov\/biblio\/1481626.","DOI":"10.2172\/1481626"},{"key":"pcbi.1008770.ref055","doi-asserted-by":"crossref","first-page":"76","DOI":"10.3389\/fninf.2017.00076","article-title":"Reproducibility vs. Replicability: A Brief History of a Confused Terminology","volume":"11","author":"HE Plesser","year":"2017","journal-title":"Front Neuroinform"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008770","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,18]],"date-time":"2021-03-18T17:31:04Z","timestamp":1616088664000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008770"}},"subtitle":[],"editor":[{"given":"Patricia M.","family":"Palagi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,3,18]]},"references-count":55,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,3,18]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1008770","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,18]]}}}