{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T18:48:35Z","timestamp":1790707715507,"version":"4.1.0"},"reference-count":43,"publisher":"Linnaeus University","license":[{"start":{"date-parts":[[2019,11,12]],"date-time":"2019-11-12T00:00:00Z","timestamp":1573516800000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MP"],"abstract":"<jats:p>\r\n\r\n\r\nScientific progress relies on the replication and reuse of research. Recent studies suggest, however, that sharing code and data does not suffice for computational reproducibility \u2014defined as the ability of researchers to reproduce \u201cpar- ticular analysis outcomes from the same data set using the same code and software\u201d (Fidler and Wilcox, 2018). To date, creating long-term computationally reproducible code has been technically challenging and time-consuming. This tutorial introduces Code Ocean, a cloud-based computational reproducibility platform that attempts to solve these problems. It does this by adapting software engineering tools, such as Docker, for easier use by scientists and scientific audiences. In this article, we first outline arguments for the importance of computational reproducibility, as well as some reasons why this is a nontrivial problem for researchers. We then provide a step-by-step guide to getting started with containers in research using Code Ocean. (Disclaimer: the authors all worked for Code Ocean at the time of this article\u2019s writing.)\r\n\r\n\r\n<\/jats:p>","DOI":"10.15626\/mp.2018.892","type":"journal-article","created":{"date-parts":[[2019,11,12]],"date-time":"2019-11-12T17:21:22Z","timestamp":1573579282000},"source":"Crossref","is-referenced-by-count":51,"title":["Computational Reproducibility via Containers in Psychology"],"prefix":"10.15626","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5401-7751","authenticated-orcid":false,"given":"April","family":"Clyburne-Sherin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xu","family":"Fei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3909-1969","authenticated-orcid":false,"given":"Seth Ariel","family":"Green","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"6393","published-online":{"date-parts":[[2019,11,12]]},"reference":[{"issue":"1","key":"ref0","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1093\/jamia\/ocx120","article-title":"Reproducible bioconductor workflows using browser-based interactive notebooks and containers","volume":"25","author":"Almugbel","year":"2017","unstructured":"Almugbel, R., Hung, L.-H., Hu, J., Almutairy, A., Ortogero, N., Tamta, Y., & Yeung, K. Y. (2017). Reproducible bioconductor workflows using browser-based interactive notebooks and containers. Journal of the American Medical Informatics Association, 25(1), 4-12.","journal-title":"Journal of the American Medical Informatics Association"},{"key":"ref1","unstructured":"Association, A. P. (2012). Ethics code updates to the publication manual. Retrieved September, 1, 2012."},{"key":"ref2","doi-asserted-by":"publisher","unstructured":"Barba, L. A. (2016). The hard road to reproducibility. Science, 354(6308), 142-142.","DOI":"10.1126\/science.354.6308.142"},{"key":"ref3","unstructured":"Barba, L. A. (2018). Terminologies for reproducible research. CoRR, abs\/1802.03311."},{"issue":"1","key":"ref4","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1145\/2723872.2723882","article-title":"An introduction to docker for reproducible research","volume":"49","author":"Boettiger","year":"2015","unstructured":"Boettiger, C. (2015). An introduction to docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71-79.","journal-title":"ACM SIGOPS Operating Systems Review"},{"key":"ref5","doi-asserted-by":"publisher","unstructured":"Boettiger, C. & Eddelbuettel, D. (2017). An introduction to rocker: docker containers for r. arXiv preprint arXiv:1710.03675.","DOI":"10.32614\/RJ-2017-065"},{"key":"ref6","unstructured":"Bogart, C., K\u00e4stner, C., & Herbsleb, J. (2015). When it breaks, it breaks. In Proc. of the workshop on software support for collaborative and global software engineering (scgse)."},{"key":"ref7","unstructured":"Chamberlain, R. & Schommer, J. (2014). Using docker to support reproducible research. DOI: https:\/\/doi. org\/10.6084\/m9. figshare, 1101910."},{"key":"ref8","unstructured":"Claerbout, J. (2011). Reproducible computational research: a history of hurdles, mostly overcome. technical report."},{"issue":"6251","key":"ref9","doi-asserted-by":"publisher","first-page":"aac4716","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Collaboration","year":"2015","unstructured":"Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.","journal-title":"Science"},{"issue":"49","key":"ref10","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1016\/j.jesp.2012.12.006","article-title":"On fraud, deceit and ethics","volume":"2","author":"Cooper","year":"2013","unstructured":"Cooper, J. (2013). On fraud, deceit and ethics. Journal of Experimental Social Psychology, 2(49), 314.","journal-title":"Journal of Experimental Social Psychology"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2008.24","article-title":"Data management challenges of data-intensive scientific workflows","author":"Deelman","year":"2008","unstructured":"Deelman, E. & Chervenak, A. (2008). Data management challenges of data-intensive scientific workflows. In Cluster computing and the grid, 2008. ccgrid'08. 8th ieee international symposium on (pp. 687- 692). IEEE.","journal-title":"In Cluster computing and the grid 2008 ccgrid 08 8th ieee international symposium on (pp"},{"issue":"4","key":"ref12","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1080\/10618600.2017.1384734","article-title":"50 years of data science","volume":"26","author":"Donoho","year":"2017","unstructured":"Donoho, D. (2017). 50 years of data science. Journal of Computational and Graphical Statistics, 26(4), 745-766.","journal-title":"Journal of Computational and Graphical Statistics"},{"key":"ref13","unstructured":"Donoho, D., Maleki, A., Rahman, I., Shahram, M., & Stodden, V. (2008). 15 years of reproducible research in computational harmonic analysis. Technical report."},{"key":"ref14","doi-asserted-by":"publisher","unstructured":"Eubank, N. (2016). Lessons from a decade of replications at the quarterly journal of political science. PS: Political Science & Politics, 49(2), 273-276.","DOI":"10.1017\/S1049096516000196"},{"key":"ref15","unstructured":"Fidler, F. & Wilcox, J. (2018). Reproducibility of scientific results. In E. N. Zalta (Ed.), The stanford encyclopedia of philosophy (Winter 2018). Metaphysics Research Lab, Stanford University."},{"issue":"1","key":"ref16","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1088868313507536","article-title":"Improving the dependability of research in personality and social psychology: recommendations for research and educational practice","volume":"18","author":"Funder","year":"2014","unstructured":"Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: recommendations for research and educational practice. Personality and Social Psychology Review, 18(1), 3-12.","journal-title":"Personality and Social Psychology Review"},{"issue":"6","key":"ref17","doi-asserted-by":"publisher","first-page":"460","DOI":"10.1511\/2014.111.460","article-title":"The statistical crisis in science","volume":"102","author":"Gelman","year":"2014","unstructured":"Gelman, A. & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460.","journal-title":"American Scientist"},{"key":"ref18","doi-asserted-by":"publisher","unstructured":"Gilad, Y. & Mizrahi-Man, O. (2015). A reanalysis of mouse encode comparative gene expression data. F1000Research,","DOI":"10.12688\/f1000research.6536.1"},{"key":"ref19","unstructured":"Grange, J., Lakens, D., Adolfi, F., Albers, C., Anvari, F., Apps, M., . . . Benning, S., et al. (2018). Justify your alpha. Nature Human Behavior."},{"key":"ref20","doi-asserted-by":"publisher","unstructured":"Gr\u00fcning, B., Rasche, E., Rebolledo-Jaramillo, B., Eberhart, C., Houwaart, T., Chilton, J., . . . Nekrutenko, A. (2016). Enhancing pre-defined workflows with ad hoc analytics using galaxy, docker and jupyter. bioRxiv, 075457.","DOI":"10.1101\/075457"},{"issue":"8","key":"ref21","doi-asserted-by":"publisher","first-page":"180448","DOI":"10.1098\/rsos.180448","article-title":"Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal cognition","volume":"5","author":"Hardwicke","year":"2018","unstructured":"Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., . . . Henry Tessler, M., et al. (2018). Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal cognition. Royal Society open science, 5(8), 180448.","journal-title":"Royal Society open science"},{"issue":"4","key":"ref22","doi-asserted-by":"publisher","first-page":"e0152686","DOI":"10.1371\/journal.pone.0152686","article-title":"Guidock: using docker containers with a common graphics user interface to address the reproducibility of research","volume":"11","author":"Hung","year":"2016","unstructured":"Hung, L.-H., Kristiyanto, D., Lee, S. B., & Yeung, K. Y. (2016). Guidock: using docker containers with a common graphics user interface to address the reproducibility of research. PloS one, 11(4), e0152686.","journal-title":"PloS one"},{"key":"ref23","unstructured":"Jonas, K. J. & Cesario, J. (2015). Guidelines for authors. Retrieved from http:\/\/www.tandf.co.uk\/journals\/ authors\/rrsp-submission-guidelines.pdf"},{"key":"ref24","doi-asserted-by":"publisher","unstructured":"Kitzes, J. (2017). Introduction. In J. Kitzes, D. Turek, & F. Deniz (Eds.), The practice of reproducible research: case studies and lessons from the dataintensive sciences. University of California Press.","DOI":"10.1525\/9780520967779"},{"key":"ref25","unstructured":"Kluyver, T., Ragan-Kelley, B., P\u00e9rez, F., Granger, B. E., Bussonnier, M., Frederic, J., . . . Corlay, S., et al. (2016). Jupyter notebooks-a publishing format for reproducible computational workflows. In Elpub (pp. 87-90)."},{"key":"ref26","unstructured":"Lindsay, D. S. (2017). Sharing data and materials in psychological science. SAGE Publications Sage CA: Los Angeles, CA."},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1177\/2378023119849803","article-title":"Successes and struggles with computational reproducibility: lessons from the fragile families challenge","author":"Liu","year":"2019","unstructured":"Liu, D. & Salganik, M. (2019). Successes and struggles with computational reproducibility: lessons from the fragile families challenge. SocArXiv.","journal-title":"SocArXiv"},{"key":"ref28","doi-asserted-by":"publisher","unstructured":"Marwick, B., Rokem, A., & Staneva, V. (2017). Assessing reproducibility. In J. Kitzes, D. Turek, & F. Deniz (Eds.), The practice of reproducible research: case studies and lessons from the data-intensive sciences. Univ of California Press.","DOI":"10.31235\/osf.io\/gne3w"},{"key":"ref29","unstructured":"Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2."},{"key":"ref30","unstructured":"Morey, R. D. & Lakens, D. (2016). Why most of psychology is statistically unfalsifiable. Submitted."},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1027\/1864-9335\/a000192","article-title":"Registered reports","author":"Nosek","year":"2014","unstructured":"Nosek, B. A. & Lakens, D. (2014). Registered reports. Hogrefe Publishing.","journal-title":"Hogrefe Publishing"},{"key":"ref32","doi-asserted-by":"publisher","unstructured":"Paluck, E. L., Green, S. A., & Green, D. P. (2018). The contact hypothesis re-evaluated. Behavioural Public Policy, 1-30.","DOI":"10.31235\/osf.io\/w2jkf"},{"key":"ref33","doi-asserted-by":"publisher","unstructured":"Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226-1227.","DOI":"10.1126\/science.1213847"},{"issue":"10","key":"ref34","doi-asserted-by":"publisher","first-page":"e1003285","DOI":"10.1371\/journal.pcbi.1003285","article-title":"Ten simple rules for reproducible computational research","volume":"9","author":"Sandve","year":"2013","unstructured":"Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS computational biology, 9(10), e1003285.","journal-title":"PLoS computational biology"},{"key":"ref35","doi-asserted-by":"publisher","unstructured":"Silver, A. (2017). Software simplified. Nature, 546(7656), 173-174.","DOI":"10.1038\/546173a"},{"issue":"11","key":"ref36","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"Simmons","year":"2011","unstructured":"Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.","journal-title":"Psychological science"},{"key":"ref37","doi-asserted-by":"publisher","unstructured":"Stodden, V. (2014). What scientific idea is ready for retirement. Edge. Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584-2589.","DOI":"10.1073\/pnas.1708290115"},{"key":"ref38","doi-asserted-by":"publisher","unstructured":"Vanpaemel, W., Vermorgen, M., Deriemaecker, L., & Storms, G. (2015). Are we wasting a good crisis? the availability of psychological research data after the storm. Collabra: Psychology, 1(1).","DOI":"10.1525\/collabra.13"},{"issue":"7","key":"ref39","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1037\/0003-066X.61.7.726","article-title":"The poor availability of psychological research data for reanalysis","volume":"61","author":"Wicherts","year":"2006","unstructured":"Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726. 9","journal-title":"American Psychologist"},{"issue":"6","key":"ref40","doi-asserted-by":"publisher","first-page":"e1005510","DOI":"10.1371\/journal.pcbi.1005510","article-title":"Good enough practices in scientific computing","volume":"13","author":"Wilson","year":"2017","unstructured":"Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017). Good enough practices in scientific computing. PLoS computational biology, 13(6), e1005510.","journal-title":"PLoS computational biology"},{"key":"ref41","doi-asserted-by":"publisher","unstructured":"Wood, B. D., M\u00fcller, R., & Brown, A. N. (2018). Push button replication: is impact evaluation evidence for international development verifiable? PloS one, 13(12), e0209416.","DOI":"10.1371\/journal.pone.0209416"},{"key":"ref42","unstructured":"Woodbridge, M. (2017). Jupyter notebooks and reproducible data science. Retrieved from https : \/ \/ markwoodbridge . com \/ 2017 \/ 03 \/ 05 \/ jupyter - reproducible-science.html"}],"container-title":["Meta-Psychology"],"original-title":[],"link":[{"URL":"https:\/\/open.lnu.se\/index.php\/metapsychology\/article\/download\/892\/2088","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/open.lnu.se\/index.php\/metapsychology\/article\/download\/892\/2088","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T13:31:00Z","timestamp":1685712660000},"score":1,"resource":{"primary":{"URL":"https:\/\/open.lnu.se\/index.php\/metapsychology\/article\/view\/892"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,12]]},"references-count":43,"URL":"https:\/\/doi.org\/10.15626\/mp.2018.892","relation":{},"ISSN":["2003-2714"],"issn-type":[{"value":"2003-2714","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,12]]}}}