{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T10:03:51Z","timestamp":1783073031836,"version":"3.54.6"},"reference-count":25,"publisher":"National Academy of Sciences","issue":"11","license":[{"start":{"date-parts":[[2018,9,12]],"date-time":"2018-09-12T00:00:00Z","timestamp":1536710400000},"content-version":"vor","delay-in-days":184,"URL":"http:\/\/www.pnas.org\/site\/aboutpnas\/licenses.xhtml"}],"funder":[{"DOI":"10.13039\/100000879","name":"Alfred P Sloan Foundation","doi-asserted-by":"crossref","award":["PG004545"],"award-info":[{"award-number":["PG004545"]}],"id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1153384"],"award-info":[{"award-number":["1153384"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2018,3,13]]},"abstract":"<jats:p>\n                    A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by (\n                    <jats:italic>i<\/jats:italic>\n                    ) requesting data and code from authors and (\n                    <jats:italic>ii<\/jats:italic>\n                    ) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal\n                    <jats:italic>Science<\/jats:italic>\n                    after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy\u2014author remission of data and code postpublication upon request\u2014an improvement over no policy, but currently insufficient for reproducibility.\n                  <\/jats:p>","DOI":"10.1073\/pnas.1708290115","type":"journal-article","created":{"date-parts":[[2018,3,12]],"date-time":"2018-03-12T15:44:40Z","timestamp":1520869480000},"page":"2584-2589","update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":268,"title":["An empirical analysis of journal policy effectiveness for computational reproducibility"],"prefix":"10.1073","volume":"115","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2015-7825","authenticated-orcid":false,"given":"Victoria","family":"Stodden","sequence":"first","affiliation":[{"name":"School of Information Sciences, University of Illinois at Urbana\u2013Champaign, Champaign, IL 61820;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Seiler","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University, New York, NY 10027"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhaokun","family":"Ma","sequence":"additional","affiliation":[{"name":"Department of Statistics, Columbia University, New York, NY 10027"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2018,3,12]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1017\/S1049096500057607","article-title":"Replication, replication","volume":"28","author":"King G","year":"1995","unstructured":"G King, Replication, replication. PS Polit Sci Polit 28, 444\u2013452 (1995).","journal-title":"PS Polit Sci Polit"},{"key":"e_1_3_4_2_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1198\/106186007X178663","article-title":"Statistical analyses and reproducible research","volume":"16","author":"Gentleman R","year":"2007","unstructured":"R Gentleman, DT Lang, Statistical analyses and reproducible research. J Comput Graphical Stat 16, 1\u201323 (2007).","journal-title":"J Comput Graphical Stat"},{"key":"e_1_3_4_3_2","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MCSE.2009.15","article-title":"Reproducible research in computational harmonic analysis","volume":"11","author":"Donoho DL","year":"2009","unstructured":"DL Donoho, A Maleki, IU Rahman, M Shahram, V Stodden, Reproducible research in computational harmonic analysis. Comput Sci Eng 11, 8\u201318 (2009).","journal-title":"Comput Sci Eng"},{"key":"e_1_3_4_4_2","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1126\/science.aah6168","article-title":"Enhancing reproducibility for computational methods","volume":"354","author":"Stodden V","year":"2016","unstructured":"V Stodden, , Enhancing reproducibility for computational methods. Science 354, 1240\u20131241 (2016).","journal-title":"Science"},{"key":"e_1_3_4_5_2","unstructured":"; National Academies of Sciences Engineering and Medicine Fostering Integrity in Research . (The National Academies Pres Washington DC. (2017)."},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1126\/science.aab2374","article-title":"Promoting an open research culture","volume":"348","author":"Nosek BA","year":"2015","unstructured":"BA Nosek, , Promoting an open research culture. Science 348, 1422\u20131425 (2015).","journal-title":"Science"},{"key":"e_1_3_4_7_2","doi-asserted-by":"crossref","first-page":"e67111","DOI":"10.1371\/journal.pone.0067111","article-title":"Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals","volume":"8","author":"Stodden V","year":"2013","unstructured":"V Stodden, P Guo, Z Ma, Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals. PLoS One 8, e67111 (2013).","journal-title":"PLoS One"},{"key":"e_1_3_4_8_2","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1126\/science.1203354","article-title":"Making data maximally available","volume":"331","author":"Hanson B","year":"2011","unstructured":"B Hanson, A Sugden, B Alberts, Making data maximally available. Science 331, 649 (2011).","journal-title":"Science"},{"key":"e_1_3_4_9_2","unstructured":"; American Association for the Advancement of Science Science journals: Editorial Policies. Available at www.sciencemag.org\/authors\/science-journals-editorial-policies. Accessed September 11 2011. (2011)."},{"key":"e_1_3_4_10_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/meet.14504701240","article-title":"Definitions of dataset in the scientific and technical literature","volume":"47","author":"Renear AH","year":"2010","unstructured":"AH Renear, S Sacchi, KM Wickett, Definitions of dataset in the scientific and technical literature. Proc Am Soc Inf Sci Technol 47, 1\u20134 (2010).","journal-title":"Proc Am Soc Inf Sci Technol"},{"key":"e_1_3_4_11_2","unstructured":"V Stodden JM Borwein DH Bailey \u201cSetting the default to reproducible\u2019 in Computational Science Research. SIAM News . Available at https:\/\/sinews.siam.org\/Aboutthe-Author\/setting-the-default-to-reproducible-in-computational-science-research. Accessed February 16 2018. (2013)."},{"key":"e_1_3_4_12_2","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1126\/science.1213847","article-title":"Reproducible research in computational science","volume":"334","author":"Peng RD","year":"2011","unstructured":"RD Peng, Reproducible research in computational science. Science 334, 1226\u20131227 (2011).","journal-title":"Science"},{"key":"e_1_3_4_13_2","unstructured":"V Stodden DH Bailey J Borwein Set the default to \u2019open\u2019. Notices of the AMS. (2013)."},{"key":"e_1_3_4_14_2","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1109\/MCSE.2009.19","article-title":"The legal framework for reproducible scientific research: Licensing and copyright","volume":"11","author":"Stodden V","year":"2009","unstructured":"V Stodden, The legal framework for reproducible scientific research: Licensing and copyright. Comput Sci Eng 11, 35\u201340 (2009).","journal-title":"Comput Sci Eng"},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/ng.295","article-title":"Repeatability of published microarray gene expression analyses","volume":"41","author":"Ioannidis JPA","year":"2008","unstructured":"JPA Ioannidis, , Repeatability of published microarray gene expression analyses. Nat Genet 41, 149\u2013155 (2008).","journal-title":"Nat Genet"},{"key":"e_1_3_4_16_2","doi-asserted-by":"crossref","first-page":"4925","DOI":"10.1111\/j.1365-294X.2012.05754.x","article-title":"Recommendations for utilizing and reporting population genetic analyses: The reproducibility of genetic clustering using the program structure","volume":"21","author":"Gilbert KJ","year":"2012","unstructured":"KJ Gilbert, , Recommendations for utilizing and reporting population genetic analyses: The reproducibility of genetic clustering using the program structure. Mol Ecol 21, 4925\u20134930 (2012).","journal-title":"Mol Ecol"},{"key":"e_1_3_4_17_2","first-page":"1","article-title":"Is economics research replicable? Sixty published papers from thirteen journals say \u201cusually not\u201d","volume":"83","author":"Chang AC","year":"2015","unstructured":"AC Chang, L Phillip, Is economics research replicable? Sixty published papers from thirteen journals say \u201cusually not\u201d. Finance Econ Discussion Ser 83, 1\u201326 (2015).","journal-title":"Finance Econ Discussion Ser"},{"key":"e_1_3_4_18_2","unstructured":"M Fuentes Reproducible research in JASA. AMSTAT News. Available at https:\/\/magazine.amstat.org\/blog\/2016\/07\/01\/jasa-reproducible16\/. Accessed February 16 2018. (2016)."},{"key":"e_1_3_4_19_2","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1017\/CBO9781107590205","volume-title":"Privacy, Big Data, and the Public Good","author":"Lane J","year":"2014","unstructured":"J Lane, V Stodden, S Bender, H Nissenbaum, Enabling reproducibility in big data research: Balancing confidentiality and scientific transparency. Privacy, Big Data, and the Public Good (Cambridge Univ Press, New York), pp. 112\u2013132 (2014)."},{"key":"e_1_3_4_20_2","doi-asserted-by":"crossref","first-page":"594","DOI":"10.1111\/jlme.12302","article-title":"Certificates of confidentiality: Protecting human subject research data in law and practice","volume":"43","author":"Wolf LE","year":"2015","unstructured":"LE Wolf, , Certificates of confidentiality: Protecting human subject research data in law and practice. J L Med Ethics 43, 594\u2013609 (2015).","journal-title":"J L Med Ethics"},{"key":"e_1_3_4_21_2","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1093\/aje\/kwx065","article-title":"Reproducing epidemiologic research and ensuring transparency","volume":"186","author":"Coughlin S","year":"2017","unstructured":"S Coughlin, Reproducing epidemiologic research and ensuring transparency. Am J Epidemiol 186, 393\u2013394 (2017).","journal-title":"Am J Epidemiol"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/B978-0-44-451875-0.00019-1","article-title":"Perturbation methods for protecting numerical data: Evolution and evaluation","volume":"28","author":"Muralidhar K","year":"2012","unstructured":"K Muralidhar, R Sarathy, Perturbation methods for protecting numerical data: Evolution and evaluation. Handbook of Statistics 28, 513\u2013531 (2012).","journal-title":"Handbook of Statistics"},{"key":"e_1_3_4_23_2","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1017\/CBO9781107590205","volume-title":"Privacy, Big Data, and the Public Good","author":"Lane J","year":"2014","unstructured":"J Lane, V Stodden, S Bender, H Nissenbaum, Differential privacy: A cryptographic approach to private data analysis. Privacy, Big Data, and the Public Good (Cambridge Univ Press, New York), pp. 296\u2013320 (2014)."},{"key":"e_1_3_4_24_2","doi-asserted-by":"crossref","DOI":"10.1145\/2602649.2602651","article-title":"A primer on provenance","volume":"12","author":"Carata L","year":"2014","unstructured":"L Carata, , A primer on provenance. ACM Queue 12 (2014).","journal-title":"ACM Queue"},{"key":"e_1_3_4_25_2","article-title":"Computing Environments for Reproducibility: Capturing the \u201cWhole Tale.\u201d","author":"Brinckman A","year":"2018","unstructured":"A Brinckman, , Computing Environments for Reproducibility: Capturing the \u201cWhole Tale.\u201d. Future Gener Comp Sy, February 10, 2018).","journal-title":"Future Gener Comp Sy"}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.pnas.org\/syndication\/doi\/10.1073\/pnas.1708290115","content-type":"unspecified","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.1708290115","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,7]],"date-time":"2022-06-07T12:56:04Z","timestamp":1654606564000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.1708290115"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,3,12]]},"references-count":25,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2018,3,13]]}},"alternative-id":["10.1073\/pnas.1708290115"],"URL":"https:\/\/doi.org\/10.1073\/pnas.1708290115","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,3,12]]},"assertion":[{"value":"2018-03-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}