{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:56:51Z","timestamp":1792792611177,"version":"4.5.0"},"reference-count":156,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2023,10,23]],"date-time":"2023-10-23T00:00:00Z","timestamp":1698019200000},"content-version":"vor","delay-in-days":31,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,9,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Computational reproducibility is a simple premise in theory, but is difficult to achieve in practice. Building upon past efforts and proposals to maximize reproducibility and rigor in bioinformatics, we present a framework called the five pillars of reproducible computational research. These include (1) literate programming, (2) code version control and sharing, (3) compute environment control, (4) persistent data sharing and (5) documentation. These practices will ensure that computational research work can be reproduced quickly and easily, long into the future. This guide is designed for bioinformatics data analysts and bioinformaticians in training, but should be relevant to other domains of study.<\/jats:p>","DOI":"10.1093\/bib\/bbad375","type":"journal-article","created":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T01:29:03Z","timestamp":1696296543000},"source":"Crossref","is-referenced-by-count":85,"title":["The five pillars of computational reproducibility: bioinformatics and beyond"],"prefix":"10.1093","volume":"24","author":[{"given":"Mark","family":"Ziemann","sequence":"first","affiliation":[{"name":"Deakin University, School of Life and Environmental Sciences , Geelong , Australia"},{"name":"Burnet Institute , Melbourne , Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierre","family":"Poulain","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Paris Cit\u00e9, CNRS, Institut Jacques Monod , Paris , France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anusuiya","family":"Bora","sequence":"additional","affiliation":[{"name":"Deakin University, School of Life and Environmental Sciences , Geelong , Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"key":"2023102311221560200_ref1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1126\/scitranslmed.aaf5027","article-title":"What does research reproducibility mean?","volume":"8","author":"Goodman","year":"2016","journal-title":"Sci Transl Med"},{"key":"2023102311221560200_ref2","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1146\/annurev-psych-020821-114157","article-title":"Replicability, robustness, and reproducibility in psychological science","volume":"73","author":"Nosek","year":"2022","journal-title":"Annu Rev Psychol"},{"key":"2023102311221560200_ref3","first-page":"17","article-title":"First results from psychology\u2019s largest reproducibility test","volume":"521","author":"Baker","year":"2015","journal-title":"Nature"},{"key":"2023102311221560200_ref4","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/483531a","article-title":"Drug development: raise standards for preclinical cancer research: drug development","volume":"483","author":"Begley","year":"2012","journal-title":"Nature"},{"key":"2023102311221560200_ref5","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/ng.295","article-title":"Repeatability of published microarray gene expression analyses","volume":"41","author":"Ioannidis","year":"2009","journal-title":"Nat Genet"},{"key":"2023102311221560200_ref6","article-title":"Data and code for reproducible research: lessons learned from the NLM reproducibility workshop","author":"Zaringhalam","year":"2020","journal-title":"Zenodo"},{"key":"2023102311221560200_ref7","article-title":"Computational reproducibility of Jupyter notebooks from biomedical publications","author":"Samuel","year":"2022"},{"key":"2023102311221560200_ref8","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1038\/s41597-022-01143-6","article-title":"A large-scale study on research code quality and execution","volume":"9","author":"Trisovic","year":"2022","journal-title":"Sci Data"},{"key":"2023102311221560200_ref9","doi-asserted-by":"crossref","first-page":"1294","DOI":"10.1038\/nm1491","article-title":"Genomic signatures to guide the use of chemotherapeutics","volume":"12","author":"Potti","year":"2006","journal-title":"Nat Med"},{"key":"2023102311221560200_ref10","first-page":"1309","article-title":"Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology","volume-title":"Ann Appl Stat","author":"Baggerly","year":"2010"},{"key":"2023102311221560200_ref11","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1038\/nm0111-135","article-title":"Retraction note: genomic signatures to guide the use of chemotherapeutics","volume":"17","author":"Potti","year":"2011","journal-title":"Nat Med"},{"key":"2023102311221560200_ref12","doi-asserted-by":"crossref","first-page":"h2559","DOI":"10.1136\/bmj.h2559","article-title":"Duke university settles lawsuits alleging that patients were harmed in chemotherapy trials","volume":"350","author":"Dyer","year":"2015","journal-title":"BMJ"},{"issue":"Suppl 4","key":"2023102311221560200_ref13","doi-asserted-by":"crossref","first-page":"S27","DOI":"10.1186\/1471-2164-11-S4-S27","article-title":"Advancing standards for bioinformatics activities: persistence, reproducibility, disambiguation and minimum information about a bioinformatics investigation (MIABi)","volume":"11","author":"Tan","year":"2010","journal-title":"BMC Genomics"},{"key":"2023102311221560200_ref14","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1126\/science.1213847","article-title":"Reproducible research in computational science","volume":"334","author":"Peng","year":"2011","journal-title":"Science"},{"key":"2023102311221560200_ref15","doi-asserted-by":"crossref","first-page":"e1003285","DOI":"10.1371\/journal.pcbi.1003285","article-title":"Ten simple rules for reproducible computational research","volume":"9","author":"Sandve","year":"2013","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref16","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/s13742-016-0135-4","article-title":"Tools and techniques for computational reproducibility","volume":"5","author":"Piccolo","year":"2016","journal-title":"Gigascience."},{"key":"2023102311221560200_ref17","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1186\/s12918-016-0288-x","article-title":"Where next for the reproducibility agenda in computational biology?","volume":"10","author":"Lewis","year":"2016","journal-title":"BMC Syst Biol"},{"key":"2023102311221560200_ref18","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1126\/science.aah6168","article-title":"Enhancing reproducibility for computational methods","volume":"354","author":"Stodden","year":"2016","journal-title":"Science"},{"key":"2023102311221560200_ref19","doi-asserted-by":"crossref","first-page":"e1005510","DOI":"10.1371\/journal.pcbi.1005510","article-title":"Good enough practices in scientific computing","volume":"13","author":"Wilson","year":"2017","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref20","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1038\/nbt.3780","article-title":"Reproducibility of computational workflows is automated using continuous analysis","volume":"35","author":"Beaulieu-Jones","year":"2017","journal-title":"Nat Biotechnol"},{"key":"2023102311221560200_ref21","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1016\/j.cels.2018.03.014","article-title":"Practical computational reproducibility in the life sciences","volume":"6","author":"Gr\u00fcning","year":"2018","journal-title":"Cell Syst"},{"key":"2023102311221560200_ref22","doi-asserted-by":"crossref","DOI":"10.1093\/gigascience\/giy077","article-title":"Experimenting with reproducibility: a case study of robustness in bioinformatics","volume":"7","author":"Kim","year":"2018","journal-title":"Gigascience"},{"key":"2023102311221560200_ref23","volume-title":"Scientific Data Analysis Pipelines and Reproducibility","author":"Akalin","year":"2018"},{"key":"2023102311221560200_ref24","first-page":"9","article-title":"Recommendations to enhance rigor and reproducibility in biomedical research","volume":"9","author":"Brito","year":"2020","journal-title":"Gigascience"},{"key":"2023102311221560200_ref25","doi-asserted-by":"crossref","first-page":"e3763","DOI":"10.5964\/qcmb.3763","article-title":"A reproducible data analysis workflow with r markdown, git, make, and docker","volume":"1","author":"Peikert","year":"2021","journal-title":"Quant Comput Methods Behav Sci"},{"key":"2023102311221560200_ref26","article-title":"Building reproducible analytical pipelines with R","author":"Rodrigues","year":"2023","journal-title":"Leanpub"},{"key":"2023102311221560200_ref27","doi-asserted-by":"crossref","first-page":"e1009651","DOI":"10.1371\/journal.pcbi.1009651","article-title":"Beyond advertising: new infrastructures for publishing integrated research objects","volume":"18","author":"DuPre","year":"2022","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref28","article-title":"Learning from reproducing computational results: introducing three principles and the reproduction package","volume":"379","author":"Krafczyk","year":"2021","journal-title":"Philos Trans A Math Phys Eng Sci"},{"key":"2023102311221560200_ref29","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pbio.1002506","article-title":"A practical guide for improving transparency and reproducibility in neuroimaging research","volume":"14","author":"Gorgolewski","year":"2016","journal-title":"PLoS Biol"},{"key":"2023102311221560200_ref30","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1109\/HICSS.1996.495416","volume-title":"Proceedings of HICSS-29: 29th Hawaii International Conference on System Sciences","author":"Panko","year":"1996"},{"key":"2023102311221560200_ref31","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1186\/1471-2105-5-80","article-title":"Mistaken identifiers: gene name errors can be introduced inadvertently when using excel in bioinformatics","volume":"5","author":"Zeeberg","year":"2004","journal-title":"BMC Bioinformatics"},{"key":"2023102311221560200_ref32","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1186\/s13059-016-1044-7","article-title":"Gene name errors are widespread in the scientific literature","volume":"17","author":"Ziemann","year":"2016","journal-title":"Genome Biol"},{"key":"2023102311221560200_ref33","doi-asserted-by":"crossref","first-page":"e1008984","DOI":"10.1371\/journal.pcbi.1008984","article-title":"Gene name errors: lessons not learned","volume":"17","author":"Abeysooriya","year":"2021","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref34","doi-asserted-by":"crossref","first-page":"692","DOI":"10.12688\/f1000research.8414.1","article-title":"Innovations in scholarly communication\u2014global survey on research tool usage","volume":"5","author":"Kramer","year":"2016","journal-title":"F1000Res"},{"key":"2023102311221560200_ref35","doi-asserted-by":"crossref","first-page":"e1009935","DOI":"10.1371\/journal.pcbi.1009935","article-title":"Urgent need for consistent standards in functional enrichment analysis","volume":"18","author":"Wijesooriya","year":"2022","journal-title":"PLoS Comput Biol"},{"issue":"50","key":"2023102311221560200_ref36","first-page":"8999","article-title":"The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update","volume":"50","author":"Community","journal-title":"Nucleic Acids Res"},{"key":"2023102311221560200_ref37","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1038\/ng0506-500","article-title":"GenePattern 2.0","volume":"38","author":"Reich","year":"2006","journal-title":"Nat Genet"},{"key":"2023102311221560200_ref38","volume-title":"Drpowell\/degust 4.1.1","author":"Powell","year":"2019"},{"key":"2023102311221560200_ref39","doi-asserted-by":"crossref","first-page":"2628","DOI":"10.1093\/bioinformatics\/btz931","article-title":"ShinyGO: a graphical gene-set enrichment tool for animals and plants","volume":"36","author":"Ge","year":"2020","journal-title":"Bioinformatics"},{"key":"2023102311221560200_ref40","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Van Der","year":"2008","journal-title":"J Mach Learn Res"},{"key":"2023102311221560200_ref41","article-title":"UMAP: uniform manifold approximation and projection for dimension reduction","author":"McInnes","year":"2018"},{"key":"2023102311221560200_ref42","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1038\/nbt.4314","article-title":"Dimensionality reduction for visualizing single-cell data using UMAP","volume":"37","author":"Becht","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2023102311221560200_ref43","doi-asserted-by":"crossref","first-page":"69","DOI":"10.3389\/fninf.2017.00069","article-title":"Re-run, repeat, reproduce, reuse, replicate: transforming code into scientific contributions","volume":"11","author":"Benureau","year":"2017","journal-title":"Front Neuroinform"},{"key":"2023102311221560200_ref44","doi-asserted-by":"crossref","first-page":"e80278","DOI":"10.1371\/journal.pone.0080278","article-title":"Quantifying reproducibility in computational biology: the case of the tuberculosis drugome","volume":"8","author":"Garijo","year":"2013","journal-title":"PloS One"},{"key":"2023102311221560200_ref45","doi-asserted-by":"crossref","first-page":"bbw020","DOI":"10.1093\/bib\/bbw020","article-title":"A review of bioinformatic pipeline frameworks","volume":"18","author":"Leipzig","year":"2016","journal-title":"Brief Bioinform"},{"key":"2023102311221560200_ref46","doi-asserted-by":"crossref","DOI":"10.1093\/gigascience\/giaa140","article-title":"Streamlining data-intensive biology with workflow systems","volume":"10","author":"Reiter","year":"2021","journal-title":"Gigascience"},{"key":"2023102311221560200_ref47","doi-asserted-by":"crossref","first-page":"2520","DOI":"10.1093\/bioinformatics\/bts480","article-title":"Snakemake\u2014a scalable bioinformatics workflow engine","volume":"28","author":"K\u00f6ster","year":"2012","journal-title":"Bioinformatics"},{"key":"2023102311221560200_ref48","doi-asserted-by":"crossref","first-page":"33","DOI":"10.12688\/f1000research.29032.1","article-title":"Sustainable data analysis with snakemake","volume":"10","author":"M\u00f6lder","year":"2021","journal-title":"F1000Res"},{"key":"2023102311221560200_ref49","doi-asserted-by":"crossref","first-page":"2959","DOI":"10.21105\/joss.02959","article-title":"The targets r package: a dynamic make-like function-oriented pipeline toolkit for reproducibility and high-performance computing","volume":"6","author":"Landau","year":"2021","journal-title":"J Open Source Softw"},{"key":"2023102311221560200_ref50","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1145\/3486897","article-title":"Methods included: standardizing computational reuse and portability with the common workflow language","volume":"65","author":"Crusoe","year":"2022","journal-title":"Commun ACM"},{"key":"2023102311221560200_ref51","article-title":"Full-stack genomics pipelining with GATK4+ WDL+ Cromwell","author":"Voss","year":"2017","journal-title":"F1000Research"},{"key":"2023102311221560200_ref52","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1038\/nbt.3820","article-title":"Nextflow enables reproducible computational workflows","volume":"35","author":"Di Tommaso","year":"2017","journal-title":"Nat Biotechnol"},{"key":"2023102311221560200_ref53","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1093\/comjnl\/27.2.97","article-title":"Literate programming","volume":"27","author":"Knuth","year":"1984","journal-title":"Comput J"},{"key":"2023102311221560200_ref54","first-page":"28","article-title":"The R journal: Sweave, part i: mixing R and LaTeX","volume":"2","author":"Leisch","year":"2002","journal-title":"R News"},{"key":"2023102311221560200_ref55","first-page":"425","volume-title":"R for Data Science","author":"Grolemund","year":"2017"},{"key":"2023102311221560200_ref56","first-page":"87","volume-title":"Positioning and Power in Academic Publishing: Players, Agents and Agendas","author":"Kluyver","year":"2016"},{"key":"2023102311221560200_ref57","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1109\/MCSE.2021.3059263","article-title":"Jupyter: thinking and storytelling with code and data","volume":"23","author":"Granger","year":"2021","journal-title":"Comput Sci Eng"},{"key":"2023102311221560200_ref58","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/d41586-018-07196-1","article-title":"Why Jupyter is data scientists\u2019 computational notebook of choice","volume":"563","author":"Perkel","year":"2018","journal-title":"Nature"},{"key":"2023102311221560200_ref59","doi-asserted-by":"crossref","first-page":"1574","DOI":"10.12688\/f1000research.9110.1","article-title":"An open RNA-Seq data analysis pipeline tutorial with an example of reprocessing data from a recent Zika virus study","volume":"5","author":"Wang","year":"2016","journal-title":"F1000Research"},{"key":"2023102311221560200_ref60","article-title":"Writing a reproducible paper with RStudio and quarto","author":"Bauer","year":"2023","journal-title":"OSF Preprints"},{"key":"2023102311221560200_ref61","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1038\/d41586-022-00563-z","article-title":"Cut the tyranny of copy-and-paste with these coding tools","volume":"603","author":"Perkel","year":"2022","journal-title":"Nature"},{"key":"2023102311221560200_ref62","author":"Posit PBC. Quarto. 2022 [cited 2023 Aug 21]. Citations & footnotes"},{"key":"2023102311221560200_ref63","doi-asserted-by":"crossref","DOI":"10.1201\/9781138359444","volume-title":"R Markdown: The Definitive Guide","author":"Xie","year":"2018"},{"key":"2023102311221560200_ref64","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1016\/j.procs.2011.04.074","article-title":"Toward executable scientific publications","volume":"4","author":"Strijkers","year":"2011","journal-title":"Procedia Comput Sci"},{"key":"2023102311221560200_ref65","article-title":"Innovation: collaborating to devise the future executable article","author":"eLife Editorial Board","year":"2017","journal-title":"eLife"},{"key":"2023102311221560200_ref66","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s42005-020-00403-4","article-title":"Creating an executable paper is a journey through open science","volume":"3","author":"Lasser","year":"2020","journal-title":"Commun Phys"},{"key":"2023102311221560200_ref67","doi-asserted-by":"crossref","first-page":"e1010193","DOI":"10.1371\/journal.pcbi.1010193","article-title":"Advancing code sharing in the computational biology community","volume":"18","author":"Cadwallader","year":"2022","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref68","article-title":"Code sharing increases citations, but remains uncommon","author":"Maitner","year":"2023","journal-title":"Res Sq"},{"key":"2023102311221560200_ref69","doi-asserted-by":"crossref","first-page":"e13933","DOI":"10.7717\/peerj.13933","article-title":"A survey of researchers\u2019 code sharing and code reuse practices, and assessment of interactive notebook prototypes","volume":"10","author":"Cadwallader","year":"2022","journal-title":"PeerJ"},{"key":"2023102311221560200_ref70","first-page":"19","volume-title":"Pro git","author":"Chacon","year":"2023","edition":"2nd"},{"key":"2023102311221560200_ref71","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/1751-0473-8-7","article-title":"Git can facilitate greater reproducibility and increased transparency in science","volume":"8","author":"Ram","year":"2013","journal-title":"Source Code Biol Med"},{"key":"2023102311221560200_ref72","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1004668","article-title":"A quick introduction to version control with git and GitHub","volume":"12","author":"Blischak","year":"2016","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref73","doi-asserted-by":"crossref","first-page":"e1004947","DOI":"10.1371\/journal.pcbi.1004947","article-title":"Ten simple rules for taking advantage of Git and GitHub","volume":"12","author":"Perez-Riverol","year":"2016","journal-title":"PLOS Comput Biol"},{"key":"2023102311221560200_ref74","volume-title":"Editorial guide","author":"JOSS Editorial Board","year":"2018"},{"key":"2023102311221560200_ref75","volume-title":"Proceedings of the 14th International Conference on Digital Preservation, iPRES 2017, Japan","author":"Di Cosmo","year":"2017"},{"key":"2023102311221560200_ref76","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/3183558","article-title":"Building the universal archive of source code. ACM, editor","volume":"61","author":"Abramatic","year":"2018","journal-title":"Commun ACM"},{"key":"2023102311221560200_ref77","first-page":"362","volume-title":"Archiving and Referencing Source Code with Software Heritage","author":"Cosmo","year":"2020"},{"key":"2023102311221560200_ref78","first-page":"87","volume-title":"Jupyter Notebooks\u2014A Publishing Format for Reproducible Computational Workflows","author":"Kluyver","year":"2016"},{"key":"2023102311221560200_ref79","article-title":"Jupyter Notebooks \u2013 a publishing format for reproducible computational workflows","volume-title":"Positioning and Power in Academic Publishing: Players, Agents and Agendas","author":"Kluyver T, Ragan-Kelley B, P\u00e9rez F, Fernando G, Brian B, Matthias F, Jonathan K, Kyle H, Jessica G, Jason C, Sylvain I, Paul A, Dami\u00e1n A, Safia W"},{"key":"2023102311221560200_ref80","article-title":"Introducing Visual Studio Code","volume-title":"Visual Studio Code Distilled","author":"Del Sole"},{"key":"2023102311221560200_ref81","article-title":"Introducing Github: A Non-Technical Guide","author":"Beer"},{"key":"2023102311221560200_ref82","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4842-5313-7_15","article-title":"Git GUI Tools","volume-title":"Beginning Git and GitHub","author":"Tsitoara"},{"key":"2023102311221560200_ref83","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1093\/bib\/bbt086","article-title":"Comparison of software packages for detecting differential expression in RNA-seq studies","volume":"16","author":"Seyednasrollah","year":"2015","journal-title":"Brief Bioinform"},{"key":"2023102311221560200_ref84","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1038\/s41597-022-01720-9","article-title":"Toward practical transparent verifiable and long-term reproducible research using Guix","volume":"9","author":"Vallet","year":"2022","journal-title":"Sci Data."},{"key":"2023102311221560200_ref85","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1093\/bib\/bbu043","article-title":"Virtual reference environments: a simple way to make research reproducible","volume":"16","author":"Hurley","year":"2015","journal-title":"Brief Bioinform"},{"key":"2023102311221560200_ref86","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/MCSE.2018.042781334","article-title":"Sharing and preserving computational analyses for posterity with encapsulator","volume":"20","author":"Pasquier","year":"2018","journal-title":"Comput Sci Eng"},{"key":"2023102311221560200_ref87","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1008316","article-title":"Ten simple rules for writing dockerfiles for reproducible data science","volume":"16","author":"N\u00fcst","year":"2020","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref88","volume-title":"What is a Container?","author":"Docker Inc","year":"2021"},{"key":"2023102311221560200_ref89","doi-asserted-by":"crossref","DOI":"10.7717\/peerj.1273","article-title":"The impact of docker containers on the performance of genomic pipelines","volume":"3","author":"Di Tommaso","year":"2015","journal-title":"PeerJ"},{"key":"2023102311221560200_ref90","first-page":"2","article-title":"Docker: lightweight linux containers for consistent development and deployment","volume":"2014","author":"Merkel","year":"2014","journal-title":"Linux J"},{"key":"2023102311221560200_ref91","article-title":"Podman\u2014A tool for managing OCI containers and pods","author":"Heon","year":"2018","journal-title":"Zenodo"},{"key":"2023102311221560200_ref92","doi-asserted-by":"crossref","first-page":"e0177459","DOI":"10.1371\/journal.pone.0177459","article-title":"Singularity: scientific containers for mobility of compute","volume":"12","author":"Kurtzer","year":"2017","journal-title":"PloS One"},{"key":"2023102311221560200_ref93","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1186\/s13059-019-1649-8","article-title":"Improving the usability and archival stability of bioinformatics software","volume":"20","author":"Mangul","year":"2019","journal-title":"Genome Biol"},{"key":"2023102311221560200_ref94","doi-asserted-by":"crossref","first-page":"e3000333","DOI":"10.1371\/journal.pbio.3000333","article-title":"Challenges and recommendations to improve the installability and archival stability of omics computational tools","volume":"17","author":"Mangul","year":"2019","journal-title":"PLoS Biol"},{"key":"2023102311221560200_ref95","doi-asserted-by":"crossref","first-page":"2580","DOI":"10.1093\/bioinformatics\/btx192","article-title":"BioContainers: an open-source and community-driven framework for software standardization","volume":"33","author":"Da","year":"2017","journal-title":"Bioinformatics"},{"key":"2023102311221560200_ref96","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1038\/d41586-023-01469-0","article-title":"The sleight-of-hand trick that can simplify scientific computing","volume":"617","author":"Perkel","year":"2023","journal-title":"Nature"},{"key":"2023102311221560200_ref97","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1109\/MCSE.2022.3165626","article-title":"Reproducibility and performance: why choose?","volume":"24","author":"Courtes","year":"2022","journal-title":"Comput Sci Eng"},{"key":"2023102311221560200_ref98","volume-title":"GNU Guix Reference Manual","author":"GNU Project Volunteers","year":"2022"},{"key":"2023102311221560200_ref99","doi-asserted-by":"crossref","first-page":"giy123","DOI":"10.1093\/gigascience\/giy123","article-title":"PiGx: reproducible genomics analysis pipelines with GNU guix","volume":"7","author":"Wurmus","year":"2018","journal-title":"Gigascience."},{"key":"2023102311221560200_ref100","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1038\/d41586-020-02462-7","article-title":"Challenge to scientists: does your ten-year-old code still run?","volume":"584","author":"Perkel","year":"2020","journal-title":"Nature"},{"key":"2023102311221560200_ref101","volume-title":"JupyterLite: Jupyter WebAssembly python","author":"Tuloup","year":"2021"},{"key":"2023102311221560200_ref102","volume-title":"Quarto-webr: Quarto Extension to Embed WebR for HTML Documents","author":"Balamuta","year":"2023"},{"key":"2023102311221560200_ref103","doi-asserted-by":"crossref","first-page":"1012","DOI":"10.1002\/jwmg.21855","article-title":"Computational reproducibility in the wildlife society\u2019s flagship journals","volume":"84","author":"Archmiller","year":"2020","journal-title":"J Wildl Manage"},{"key":"2023102311221560200_ref104","doi-asserted-by":"crossref","first-page":"5","DOI":"10.7554\/eLife.16800","article-title":"How open science helps researchers succeed","volume":"5","author":"McKiernan","year":"2016","journal-title":"Elife"},{"key":"2023102311221560200_ref105","author":"United Nations Educational, Scientific and Cultural Organization (UNESCO) recommendation on open science","year":"2021"},{"key":"2023102311221560200_ref106","doi-asserted-by":"crossref","DOI":"10.1038\/sdata.2016.18","article-title":"The FAIR guiding principles for scientific data management and stewardship","volume":"3","author":"Wilkinson","year":"2016","journal-title":"Sci Data"},{"key":"2023102311221560200_ref107","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s13041-020-0552-2","article-title":"No raw data, no science: another possible source of the reproducibility crisis","volume":"13","author":"Miyakawa","year":"2020","journal-title":"Mol Brain"},{"key":"2023102311221560200_ref108","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1093\/biostatistics\/kxq028","article-title":"An invitation to reproducible computational research","volume":"11","author":"Donoho","year":"2010","journal-title":"Biostatistics"},{"key":"2023102311221560200_ref109","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.jclinepi.2022.05.019","article-title":"Many researchers were not compliant with their published data sharing statement: A mixed-methods study","volume":"150","author":"Gabelica","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"2023102311221560200_ref110","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1038\/ng1201-365","article-title":"Minimum information about a microarray experiment (MIAME)-toward standards for microarray data","volume":"29","author":"Brazma","year":"2001","journal-title":"Nat Genet"},{"key":"2023102311221560200_ref111","volume-title":"Management of Data and Information in Research: A Guide Supporting the Australian Code for the Responsible Conduct of Research","author":"National Health and Medical Research Council","year":"2019"},{"key":"2023102311221560200_ref112","author":"Office of The Director","year":"2023"},{"key":"2023102311221560200_ref113","doi-asserted-by":"crossref","first-page":"k510","DOI":"10.1136\/bmj.k510","article-title":"Data sharing in medical research","volume":"360","author":"Packer","year":"2018","journal-title":"BMJ"},{"key":"2023102311221560200_ref114","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1126\/science.1203354","article-title":"Making data maximally available","volume":"331","author":"Hanson","year":"2011","journal-title":"Science"},{"key":"2023102311221560200_ref115","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1186\/s13059-020-02010-6","article-title":"Open data in a deeply connected world","volume":"21","author":"Cheifet","year":"2020","journal-title":"Genome Biol"},{"key":"2023102311221560200_ref116","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1038\/s41576-020-0257-5","article-title":"Responsible, practical genomic data sharing that accelerates research","volume":"21","author":"Byrd","year":"2020","journal-title":"Nat Rev Genet"},{"key":"2023102311221560200_ref117","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1159\/000294150","article-title":"Assessing the privacy risks of data sharing in genomics","volume":"14","author":"Heeney","year":"2011","journal-title":"Public Health Genomics"},{"key":"2023102311221560200_ref118","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1126\/science.1229566","article-title":"Identifying personal genomes by surname inference","volume":"339","author":"Gymrek","year":"2013","journal-title":"Science"},{"key":"2023102311221560200_ref119","doi-asserted-by":"crossref","first-page":"100029","DOI":"10.1016\/j.xgen.2021.100029","article-title":"GA4GH: international policies and standards for data sharing across genomic research and healthcare","volume":"1","author":"Rehm","year":"2021","journal-title":"Cell Genom"},{"key":"2023102311221560200_ref120","doi-asserted-by":"crossref","first-page":"100032","DOI":"10.1016\/j.xgen.2021.100032","article-title":"International federation of genomic medicine databases using GA4GH standards","volume":"1","author":"Thorogood","year":"2021","journal-title":"Cell Genom"},{"key":"2023102311221560200_ref121","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1093\/nar\/30.1.207","article-title":"Gene expression omnibus: NCBI gene expression and hybridization array data repository","volume":"30","author":"Edgar","year":"2002","journal-title":"Nucleic Acids Res"},{"key":"2023102311221560200_ref122","doi-asserted-by":"crossref","first-page":"D54","DOI":"10.1093\/nar\/gkr854","article-title":"International Nucleotide Sequence Database Collaboration. The sequence read archive: explosive growth of sequencing data","volume":"40","author":"Kodama","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2023102311221560200_ref123","doi-asserted-by":"crossref","first-page":"3537","DOI":"10.1002\/pmic.200401303","article-title":"PRIDE: the proteomics identifications database","volume":"5","author":"Martens","year":"2005","journal-title":"Proteomics"},{"key":"2023102311221560200_ref124","author":"Bonaretti","year":"2019","journal-title":"Two real use cases of FAIR maturity indicators in the life sciences"},{"key":"2023102311221560200_ref125","article-title":"FAIR enough? A perspective on the status of nucleotide sequence data and metadata on public archives","author":"Hassenr\u00fcck","year":"2021","journal-title":"bioRxiv"},{"key":"2023102311221560200_ref126","doi-asserted-by":"crossref","first-page":"e78080","DOI":"10.1371\/journal.pone.0078080","article-title":"Making research data repositories visible: the re3data.org registry","volume":"8","author":"Pampel","year":"2013","journal-title":"PloS One"},{"key":"2023102311221560200_ref127","doi-asserted-by":"crossref","first-page":"e1002295","DOI":"10.1371\/journal.pbio.1002295","article-title":"Public data archiving in ecology and evolution: how well are we doing?","volume":"13","author":"Roche","year":"2015","journal-title":"PLoS Biol"},{"issue":"Suppl 14","key":"2023102311221560200_ref128","doi-asserted-by":"crossref","first-page":"S5","DOI":"10.1186\/1471-2105-14-S14-S5","article-title":"A cross disciplinary study of link decay and the effectiveness of mitigation techniques","volume":"14","author":"Hennessey","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023102311221560200_ref129","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.anbehav.2014.09.025","article-title":"Forgotten treasures: the fate of data in animal behaviour studies","volume":"98","author":"Caetano","year":"2014","journal-title":"Anim Behav"},{"key":"2023102311221560200_ref130","doi-asserted-by":"crossref","first-page":"e1002375","DOI":"10.1371\/journal.pcbi.1002375","article-title":"Ten years of pathway analysis: current approaches and outstanding challenges","volume":"8","author":"Khatri","year":"2012","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref131","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1038\/nrg3305","article-title":"Next-generation sequencing data interpretation: enhancing reproducibility and accessibility","volume":"13","author":"Nekrutenko","year":"2012","journal-title":"Nat Rev Genet"},{"key":"2023102311221560200_ref132","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1093\/bib\/bbz124","article-title":"Current RNA-seq methodology reporting limits reproducibility","volume":"22","author":"Simoneau","year":"2021","journal-title":"Brief Bioinform"},{"key":"2023102311221560200_ref133","article-title":"MDAR: towards minimum reporting standards for life scientists","author":"Mellor","year":"2022","journal-title":"Open Science Framework"},{"key":"2023102311221560200_ref134","doi-asserted-by":"crossref","first-page":"e1007007","DOI":"10.1371\/journal.pcbi.1007007","article-title":"Ten simple rules for writing and sharing computational analyses in Jupyter notebooks","volume":"15","author":"Rule","year":"2019","journal-title":"PLoS Comput Biol"},{"key":"2023102311221560200_ref135","volume-title":"A Recipe for Extremely Reproducible Enrichment Analysis v2. protocols.io","author":"Ziemann","year":"2023"},{"key":"2023102311221560200_ref136","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1038\/s41597-023-02463-x","article-title":"Making biomedical research software FAIR: actionable step-by-step guidelines with a user-support tool","volume":"10","author":"Patel","year":"2023","journal-title":"Sci Data"},{"key":"2023102311221560200_ref137","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1126\/science.354.6308.142","article-title":"The hard road to reproducibility","volume":"354","author":"Barba","year":"2016","journal-title":"Science"},{"key":"2023102311221560200_ref138","article-title":"The Turing Way: a handbook for reproducible data science","author":"The Turing Way Community","year":"2019","journal-title":"Zenodo"},{"key":"2023102311221560200_ref139","doi-asserted-by":"crossref","first-page":"e839","DOI":"10.7717\/peerj-cs.839","article-title":"Improving bioinformatics software quality through incorporation of software engineering practices","volume":"8","author":"Noor","year":"2022","journal-title":"PeerJ Comput Sci"},{"key":"2023102311221560200_ref140","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.12688\/f1000research.9315.1","article-title":"Unit testing, model validation, and biological simulation","volume":"5","author":"Sarma","year":"2016","journal-title":"F1000Res"},{"key":"2023102311221560200_ref141","doi-asserted-by":"crossref","first-page":"5","DOI":"10.32614\/RJ-2011-002","article-title":"Testthat: get started with testing","volume":"3","author":"Wickham","year":"2011","journal-title":"The R Journal"},{"key":"2023102311221560200_ref142","volume-title":"Pytest 7.4","author":"Krekel","year":"2004"},{"key":"2023102311221560200_ref143","doi-asserted-by":"crossref","first-page":"1749","DOI":"10.12688\/f1000research.20843.1","article-title":"Creating and sharing reproducible research code the workflowr way","volume":"8","author":"Blischak","year":"2019","journal-title":"F1000Res"},{"key":"2023102311221560200_ref144","doi-asserted-by":"crossref","first-page":"e1002456","DOI":"10.1371\/journal.pbio.1002456","article-title":"Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency","volume":"14","author":"Kidwell","year":"2016","journal-title":"PLoS Biol"},{"key":"2023102311221560200_ref145","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1093\/gigascience\/giad031","article-title":"A workflow reproducibility scale for automatic validation of biological interpretation results","volume":"12","author":"Suetake","year":"2022","journal-title":"Gigascience"},{"key":"2023102311221560200_ref146","doi-asserted-by":"crossref","DOI":"10.1038\/s41562-016-0021","article-title":"A manifesto for reproducible science","volume":"1","author":"Munaf\u00f2","year":"2017","journal-title":"Nat Hum Behav"},{"key":"2023102311221560200_ref147","doi-asserted-by":"crossref","DOI":"10.1186\/s13104-021-05875-3","article-title":"Reproducibility and research integrity: the role of scientists and institutions","volume":"14","author":"Diaba-Nuhoho","year":"2021","journal-title":"BMC Res Notes"},{"key":"2023102311221560200_ref148","doi-asserted-by":"crossref","first-page":"e0183591","DOI":"10.1371\/journal.pone.0183591","article-title":"A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review","volume":"12","author":"Han","year":"2017","journal-title":"PloS One"},{"key":"2023102311221560200_ref149","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1093\/bib\/bbs078","article-title":"Comparability and reproducibility of biomedical data","volume":"14","author":"Huang","year":"2013","journal-title":"Brief Bioinform"},{"key":"2023102311221560200_ref150","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1038\/545387a","article-title":"Publish houses of brick, not mansions of straw","volume":"545","author":"Kaelin","year":"2017","journal-title":"Nature"},{"key":"2023102311221560200_ref151","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1186\/s41073-021-00113-7","article-title":"Strengthening the incentives for responsible research practices in Australian health and medical research funding","volume":"6","author":"Diong","year":"2021","journal-title":"Res Integr Peer Rev"},{"key":"2023102311221560200_ref152","article-title":"How bibliometrics and school rankings reward unreliable science","volume":"1887","author":"Oransky","year":"2023","journal-title":"BMJ"},{"key":"2023102311221560200_ref153","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1126\/scitranslmed.aaf5003","article-title":"An incentive-based approach for improving data reproducibility","volume":"8","author":"Rosenblatt","year":"2016","journal-title":"Sci Transl Med"},{"key":"2023102311221560200_ref154","doi-asserted-by":"crossref","DOI":"10.17226\/25104","volume-title":"Data Science for Undergraduates: Opportunities and Options","author":"Committee on Envisioning the Data Science Discipline: The Undergraduate Perspective, Computer Science and Telecommunications Board, Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Division on Engineering and Physical Sciences, Board on Science Education","year":"2018"},{"key":"2023102311221560200_ref155","article-title":"Optimizing short-format training: an international consensus on effective, inclusive, and career-spanning professional development in the life sciences and beyond","author":"Williams","year":"2023","journal-title":"bioRxiv"},{"key":"2023102311221560200_ref156","doi-asserted-by":"crossref","first-page":"62","DOI":"10.12688\/f1000research.3-62.v1","article-title":"Software carpentry: lessons learned","volume":"3","author":"Wilson","year":"2014","journal-title":"F1000Res"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/6\/bbad375\/52346129\/bbad375.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/24\/6\/bbad375\/52346129\/bbad375.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,23]],"date-time":"2023-10-23T07:23:27Z","timestamp":1698045807000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbad375\/7326135"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,22]]},"references-count":156,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,9,22]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbad375","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,11,1]]},"published":{"date-parts":[[2023,9,22]]},"article-number":"bbad375"}}