{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T05:12:08Z","timestamp":1790831528261,"version":"4.1.0"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,6,23]],"date-time":"2020-06-23T00:00:00Z","timestamp":1592870400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1839010, 1941443"],"award-info":[{"award-number":["1839010, 1941443"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,6,23]]},"DOI":"10.1145\/3391800.3398172","type":"proceedings-article","created":{"date-parts":[[2020,6,16]],"date-time":"2020-06-16T19:03:55Z","timestamp":1592334235000},"page":"9-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Beyond Open Data: A Model for Linking Digital Artifacts to Enable Reproducibility of Scientific Claims"],"prefix":"10.1145","author":[{"given":"Victoria","family":"Stodden","sequence":"first","affiliation":[{"name":"University of Illinois Urbana-Champaign, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,6,23]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2016. RDA and ICSU-WDS Announce the Scholix Framework for Linking Data and Literature. https:\/\/www.icsu-wds.org\/news\/news-archive\/rda-and-icsu-wds-announce-the-scholix-framework-for-linking-data-and-literature  2016. RDA and ICSU-WDS Announce the Scholix Framework for Linking Data and Literature. https:\/\/www.icsu-wds.org\/news\/news-archive\/rda-and-icsu-wds-announce-the-scholix-framework-for-linking-data-and-literature"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2019.2949413"},{"key":"e_1_3_2_1_3_1","volume-title":"Privacy on the Books and on the Ground. Stan. L. Rev.","volume":"63","author":"Bamberger K. A.","year":"2010","unstructured":"K. A. Bamberger and D. K. Mulligan . 2010 . Privacy on the Books and on the Ground. Stan. L. Rev. , Vol. 63 ( 2010 ). K. A. Bamberger and D. K. Mulligan. 2010. Privacy on the Books and on the Ground. Stan. L. Rev., Vol. 63 (2010)."},{"key":"e_1_3_2_1_4_1","volume-title":"Scholarship in the Digital Age","author":"Borgman C.","unstructured":"C. Borgman . 2007. Scholarship in the Digital Age . MIT Press . C. Borgman. 2007. Scholarship in the Digital Age .MIT Press."},{"key":"e_1_3_2_1_5_1","volume-title":"Little Data","author":"Borgman C.","unstructured":"C. Borgman . 2015. Big Data , Little Data , No Data: Scholarship in the Networked World .MIT Press . C. Borgman. 2015. Big Data, Little Data, No Data: Scholarship in the Networked World .MIT Press."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.2015.145052010096"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.12.029"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002235"},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Chard K.","unstructured":"K. Chard , N. Gaffney , M. B. Jones , K. Kowalik , B. Lud\"ascher, J. Nabrzyski , V. Stodden , I. Taylor , M. J. Turk , and C. Willis . 2019. Implementing Computational Reproducibility in the Whole Tale Environment . In Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems ( Phoenix, AZ, USA) (P-RECS '19). ACM, New York, NY, USA, 17--22. https:\/\/doi.org\/10.1145\/3322790.3330594 10.1145\/3322790.3330594 K. Chard, N. Gaffney, M. B. Jones, K. Kowalik, B. Lud\"ascher, J. Nabrzyski, V. Stodden, I. Taylor, M. J. Turk, and C. Willis. 2019. Implementing Computational Reproducibility in the Whole Tale Environment. In Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems (Phoenix, AZ, USA) (P-RECS '19). ACM, New York, NY, USA, 17--22. https:\/\/doi.org\/10.1145\/3322790.3330594"},{"key":"e_1_3_2_1_10_1","article-title":"The Right to Quantitative Privacy","volume":"98","author":"Citron D.","year":"2013","unstructured":"D. Citron and D. Gray . 2013 . The Right to Quantitative Privacy . Minnesota Law Review , Vol. 98 , 62 (2013). D. Citron and D. Gray. 2013. The Right to Quantitative Privacy. Minnesota Law Review, Vol. 98, 62 (2013).","journal-title":"Minnesota Law Review"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.acalib.2014.04.010"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2009.15"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1198\/106186007X178663"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5860\/lrts.52n1.18"},{"key":"e_1_3_2_1_15_1","unstructured":"D. Irving. [n.d.]. Best practices for scientific software. https:\/\/software.ac.uk\/blog\/2017--11--29-best-practices-scientific-software  D. Irving. [n.d.]. Best practices for scientific software. https:\/\/software.ac.uk\/blog\/2017--11--29-best-practices-scientific-software"},{"key":"e_1_3_2_1_16_1","volume-title":"Two Bits: The Cultural Significance of Software","author":"Kelty C.","unstructured":"C. Kelty . 2008. Two Bits: The Cultural Significance of Software . Duke University Press . C. Kelty. 2008. Two Bits: The Cultural Significance of Software .Duke University Press."},{"key":"e_1_3_2_1_17_1","article-title":"The digital dark ages? Challenges in the preservation of electronic information","volume":"17","author":"Kuny T.","year":"1998","unstructured":"T. Kuny . 1998 . The digital dark ages? Challenges in the preservation of electronic information . International Preservation News , Vol. 17 , 8 (1998). T. Kuny. 1998. The digital dark ages? Challenges in the preservation of electronic information. International Preservation News, Vol. 17, 8 (1998).","journal-title":"International Preservation News"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"J. Lane V. Stodden S. Bender and H. Nissenbaum. 2014. Privacy big data and the public good: Frameworks for engagement. Cambridge University Press.  J. Lane V. Stodden S. Bender and H. Nissenbaum. 2014. Privacy big data and the public good: Frameworks for engagement. Cambridge University Press.","DOI":"10.1017\/CBO9781107590205"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-014-1238-2"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23061"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-016-0021"},{"key":"e_1_3_2_1_22_1","volume-title":"Knowledge Infrastructures: Intellectual Frameworks and Research Challenges.","author":"Edwards P. N.","year":"2013","unstructured":"Edwards P. N. , Jackson S. J., Chalmers M. K. , Bowker G. C., Borgman C. L. , Ribes D., Burton M., and Calvert S . 2013 . Knowledge Infrastructures: Intellectual Frameworks and Research Challenges. Ann Arbor : Deep Blue ( 2013). Edwards P. N., Jackson S. J., Chalmers M. K., Bowker G. C., Borgman C. L., Ribes D., Burton M., and Calvert S. 2013. Knowledge Infrastructures: Intellectual Frameworks and Research Challenges. Ann Arbor: Deep Blue (2013)."},{"key":"e_1_3_2_1_23_1","unstructured":"Nature. 2020. Reporting standards and availability of data materials code and protocols. https:\/\/www.nature.com\/nature-research\/editorial-policies\/reporting-standards  Nature. 2020. Reporting standards and availability of data materials code and protocols. https:\/\/www.nature.com\/nature-research\/editorial-policies\/reporting-standards"},{"key":"e_1_3_2_1_24_1","volume-title":"Science","volume":"348","author":"Nosek B. A.","year":"2015","unstructured":"B. A. Nosek , G. Alter , G. C. Banks , D. Borsboom , S. D. Bowman , S. J. Breckler , S. Buck , C. D. Chambers , G. Chin , G. Christensen , M. Contestabile , A. Dafoe , E. Eich , J. Freese , R. Glennerster , D. Goroff , D. P. Green , B. Hesse , M. Humphreys , J. Ishiyama , D. Karlan , A. Kraut , A. Lupia , P. Mabry , T. Madon , N. Malhotra , E. Mayo-Wilson , M. McNutt , E. Miguel , E. Levy Paluck , U. Simonsohn , C. Soderberg , B. A. Spellman , J. Turitto , G. VandenBos , S. Vazire , E. J. Wagenmakers , R. Wilson , and T. Yarkoni . 2015. Promoting an open research culture . Science , Vol. 348 , 6242 ( 2015 ), 1422--1425. https:\/\/doi.org\/10.1126\/science.aab2374 10.1126\/science.aab2374 B. A. Nosek, G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, M. Contestabile, A. Dafoe, E. Eich, J. Freese, R. Glennerster, D. Goroff, D. P. Green, B. Hesse, M. Humphreys, J. Ishiyama, D. Karlan, A. Kraut, A. Lupia, P. Mabry, T. Madon, N. Malhotra, E. Mayo-Wilson, M. McNutt, E. Miguel, E. Levy Paluck, U. Simonsohn, C. Soderberg, B. A. Spellman, J. Turitto, G. VandenBos, S. Vazire, E. J. Wagenmakers, R. Wilson, and T. Yarkoni. 2015. Promoting an open research culture. Science, Vol. 348, 6242 (2015), 1422--1425. https:\/\/doi.org\/10.1126\/science.aab2374"},{"key":"e_1_3_2_1_25_1","unstructured":"Association of Computing Machinery. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging  Association of Computing Machinery. 2018. Artifact Review and Badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging"},{"key":"e_1_3_2_1_26_1","volume-title":"Reproducibility and Replicability in Science","author":"National Academies of Sciences Engineering and Medicine. 2019.","unstructured":"National Academies of Sciences Engineering and Medicine. 2019. Reproducibility and Replicability in Science . The National Academies Press , Washington, DC . https:\/\/doi.org\/10.17226\/25303 10.17226\/25303 National Academies of Sciences Engineering and Medicine. 2019. Reproducibility and Replicability in Science. The National Academies Press, Washington, DC. https:\/\/doi.org\/10.17226\/25303"},{"key":"e_1_3_2_1_27_1","article-title":"Open Science Framework (OSF)","volume":"105","author":"Open Science","year":"2017","unstructured":"Open Science Framework (OSF). 2017 . Open Science Framework (OSF) . Journal of the Medical Library Association : JMLA , Vol. 105 , 2 (04 2017), 203--206. https:\/\/doi.org\/10.5195\/jmla.2017.88 10.5195\/jmla.2017.88 Open Science Framework (OSF). 2017. Open Science Framework (OSF). Journal of the Medical Library Association : JMLA, Vol. 105, 2 (04 2017), 203--206. https:\/\/doi.org\/10.5195\/jmla.2017.88","journal-title":"Journal of the Medical Library Association : JMLA"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1213847"},{"key":"e_1_3_2_1_29_1","unstructured":"Science. 2020. Science Journals: editorial policies. https:\/\/www.sciencemag.org\/authors\/science-journals-editorial-policies#data-deposition  Science. 2020. Science Journals: editorial policies. https:\/\/www.sciencemag.org\/authors\/science-journals-editorial-policies#data-deposition"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(76)90005-4"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2009.19"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.22323\/2.09010205"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.1550193"},{"key":"e_1_3_2_1_34_1","volume-title":"Opening Science: The Evolving Guide on How the Internet is Changing Research","author":"Stodden V.","unstructured":"V. Stodden . 2014a. Intellectual Property and Computational Science . In Opening Science: The Evolving Guide on How the Internet is Changing Research , Collaboration and Scholarly Publishing, S. Bartling and S. Friesike (Eds.). Springer International Publishing , 225--235. https:\/\/doi.org\/10.1007\/978--3--319-00026--8_15 10.1007\/978--3--319-00026--8_15 V. Stodden. 2014a. Intellectual Property and Computational Science. In Opening Science: The Evolving Guide on How the Internet is Changing Research, Collaboration and Scholarly Publishing, S. Bartling and S. Friesike (Eds.). Springer International Publishing, 225--235. https:\/\/doi.org\/10.1007\/978--3--319-00026--8_15"},{"key":"e_1_3_2_1_35_1","volume-title":"What Computational Scientists Need to Know about Intellectual Property Law: A Primer","author":"Stodden V.","unstructured":"V. Stodden . 2014b. What Computational Scientists Need to Know about Intellectual Property Law: A Primer . In Implementing Reproducible Research, V. Stodden, F. Leisch, and R. D. Peng (Eds.). CRC Press . V. Stodden. 2014b. What Computational Scientists Need to Know about Intellectual Property Law: A Primer. In Implementing Reproducible Research, V. Stodden, F. Leisch, and R. D. Peng (Eds.). CRC Press."},{"key":"e_1_3_2_1_36_1","volume-title":"PLOS ONE","volume":"8","author":"Stodden V.","year":"2013","unstructured":"V. Stodden , P. Guo , and Z. Ma . 2013. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals . PLOS ONE , Vol. 8 , 6 (21 June 2013 ), e67111. https:\/\/doi.org\/10.1371\/journal.pone.0067111 10.1371\/journal.pone.0067111 V. Stodden, P. Guo, and Z. Ma. 2013. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLOS ONE, Vol. 8, 6 (21 June 2013), e67111. https:\/\/doi.org\/10.1371\/journal.pone.0067111"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"V. Stodden F. Leisch and R. D. Peng. 2014. Implementing Reproducible Research .CRC Press.  V. Stodden F. Leisch and R. D. Peng. 2014. Implementing Reproducible Research .CRC Press.","DOI":"10.1201\/b16868"},{"key":"e_1_3_2_1_38_1","volume-title":"Science","volume":"354","author":"Stodden V.","year":"2016","unstructured":"V. Stodden , M. McNutt , D. H. Bailey , E. Deelman , Y. Gil , B. Hanson , M. A. Heroux , J. P.A. Ioannidis , and M. Taufer . 2016. Enhancing reproducibility for computational methods . Science , Vol. 354 , 6317 ( 2016 ), 1240--1241. https:\/\/doi.org\/10.1126\/science.aah6168 10.1126\/science.aah6168 V. Stodden, M. McNutt, D. H. Bailey, E. Deelman, Y. Gil, B. Hanson, M. A. Heroux, J. P.A. Ioannidis, and M. Taufer. 2016. Enhancing reproducibility for computational methods. Science, Vol. 354, 6317 (2016), 1240--1241. https:\/\/doi.org\/10.1126\/science.aah6168"},{"key":"#cr-split#-e_1_3_2_1_39_1.1","doi-asserted-by":"crossref","unstructured":"V. Stodden and S. Miguez. 2013. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. JORS (Sep 2013). https:\/\/doi.org\/10.2139\/ssrn.2322276 10.2139\/ssrn.2322276","DOI":"10.2139\/ssrn.2322276"},{"key":"#cr-split#-e_1_3_2_1_39_1.2","doi-asserted-by":"crossref","unstructured":"V. Stodden and S. Miguez. 2013. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. JORS (Sep 2013). https:\/\/doi.org\/10.2139\/ssrn.2322276","DOI":"10.2139\/ssrn.2322276"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708290115"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.3163\/1536-5050.103.3.011"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0067332"}],"event":{"name":"HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing","location":"Stockholm Sweden","acronym":"HPDC '20","sponsor":["University of Arizona University of Arizona","SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing","SIGARCH ACM Special Interest Group on Computer Architecture"]},"container-title":["Proceedings of the 3rd International Workshop on Practical Reproducible Evaluation of Computer Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3391800.3398172","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3391800.3398172","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:41:41Z","timestamp":1750185701000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3391800.3398172"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,23]]},"references-count":43,"alternative-id":["10.1145\/3391800.3398172","10.1145\/3391800"],"URL":"https:\/\/doi.org\/10.1145\/3391800.3398172","relation":{},"subject":[],"published":{"date-parts":[[2020,6,23]]},"assertion":[{"value":"2020-06-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}