{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T03:39:02Z","timestamp":1768016342893,"version":"3.49.0"},"reference-count":49,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2019,4,8]],"date-time":"2019-04-08T00:00:00Z","timestamp":1554681600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100006206","name":"Office of Biological and Environmental Research","doi-asserted-by":"crossref","award":["17- SC-20-SC"],"award-info":[{"award-number":["17- SC-20-SC"]}],"id":[{"id":"10.13039\/100006206","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2019,9]]},"abstract":"<jats:p> We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics. We discuss two use cases: scientific reproducibility of results in the Energy Exascale Earth System Model (E3SM\u2014previously ACME) and performance reproducibility in molecular dynamics workflows on HPC platforms. To capture and persist the provenance and performance data of these workflows, we have designed and developed the Chimbuko and ProvEn frameworks. Chimbuko captures provenance and enables detailed single workflow performance analysis. ProvEn is a hybrid, queryable system for storing and analyzing the provenance and performance metrics of multiple runs in workflow performance analysis campaigns. Workflow provenance and performance data output from Chimbuko can be visualized in a dynamic, multilevel visualization providing overview and zoom-in capabilities for areas of interest. Provenance and related performance data ingested into ProvEn is queryable and can be used to reproduce runs. Our provenance-based approach highlights challenges in extracting information and gaps in the information collected. It is agnostic to the type of provenance data it captures so that both the reproducibility of scientific results and that of performance can be explored with our tools. <\/jats:p>","DOI":"10.1177\/1094342019839124","type":"journal-article","created":{"date-parts":[[2019,4,9]],"date-time":"2019-04-09T03:09:10Z","timestamp":1554779350000},"page":"763-776","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":17,"title":["Computational reproducibility of scientific workflows at extreme scales"],"prefix":"10.1177","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2120-6521","authenticated-orcid":false,"given":"Line","family":"Pouchard","sequence":"first","affiliation":[{"name":"Brookhaven National Laboratory, Upton, NY, USA"}]},{"given":"Sterling","family":"Baldwin","sequence":"additional","affiliation":[{"name":"Lawrence Livermore National Laboratory, Livermore, CA, USA"}]},{"given":"Todd","family":"Elsethagen","sequence":"additional","affiliation":[{"name":"Pacific Northwest National Laboratory, Richland, WA, USA"}]},{"given":"Shantenu","family":"Jha","sequence":"additional","affiliation":[{"name":"Brookhaven National Laboratory, Upton, NY, USA"}]},{"given":"Bibi","family":"Raju","sequence":"additional","affiliation":[{"name":"Pacific Northwest National Laboratory, Richland, WA, USA"}]},{"given":"Eric","family":"Stephan","sequence":"additional","affiliation":[{"name":"Pacific Northwest National Laboratory, Richland, WA, USA"}]},{"given":"Li","family":"Tang","sequence":"additional","affiliation":[{"name":"Los Alamos National Laboratory, Los Alamos, NM, USA"}]},{"given":"Kerstin Kleese","family":"Van Dam","sequence":"additional","affiliation":[{"name":"Brookhaven National Laboratory, Upton, NY, USA"}]}],"member":"179","published-online":{"date-parts":[[2019,4,8]]},"reference":[{"key":"bibr1-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/HUST.2014.6"},{"key":"bibr2-1094342019839124","author":"Bechhofer S","year":"2010","journal-title":"Nature Proceedings"},{"key":"bibr3-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723882"},{"key":"bibr4-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-14325-5_43"},{"key":"bibr5-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465269"},{"key":"bibr6-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2013.07.002"},{"key":"bibr7-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2015.09.001"},{"key":"bibr8-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2012.41"},{"key":"bibr9-1094342019839124","first-page":"57","volume-title":"Implementing Reproducible Research","author":"Davison AP","year":"2014"},{"key":"bibr10-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/ARITH.2013.43"},{"key":"bibr11-1094342019839124","volume-title":"Workshop on evaluation methods for machine learning at the international conference on machine learning (ICML)","author":"Drummond C","year":"2009"},{"key":"bibr12-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/NYSDS.2016.7747819"},{"key":"bibr13-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807623"},{"key":"bibr14-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1002\/2015EA000136"},{"key":"bibr15-1094342019839124","first-page":"3","volume-title":"International joint conference on knowledge discovery, knowledge engineering, and knowledge management","author":"Goble C","year":"2011"},{"key":"bibr16-1094342019839124","first-page":"596","volume-title":"European conference on parallel processing","author":"Gramoli V","year":"2016"},{"key":"bibr17-1094342019839124","doi-asserted-by":"publisher","DOI":"10.2172\/1481626"},{"key":"bibr55-1094342019839124","doi-asserted-by":"crossref","unstructured":"Huang H, Qin H, Yoo S, et al. (2014) Physics-Based Anomaly Detection Defined on Manifold Space. ACM Transactions on Knowledge Discovery from Data (TKDD) 9(2): 1\u201339. DOI: https:\/\/doi.org\/10.1145\/2641574","DOI":"10.1145\/2641574"},{"key":"bibr19-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/2949550.2949560"},{"key":"bibr20-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1007\/11846802_44"},{"key":"bibr21-1094342019839124","unstructured":"Hunold S, Tr\u00e4ff JL (2013) On the state and importance of reproducible experimental research in parallel computing. arXiv preprint arXiv:1308.3648."},{"key":"bibr22-1094342019839124","unstructured":"James D, Wilkins-Diehr N, Stodden V, et al. (2014) Standing together for reproducibility in large-scale computing: report on reproducibility @ XSEDE. arXiv preprint arXiv:1412.5557."},{"key":"bibr23-1094342019839124","unstructured":"Johansen H, McInnes L, Bernholdt D, et al. (2014) Software Productivity for Extreme-Scale Science (2014). DOE ASCR Workshop Report, pp. 13\u201314."},{"key":"bibr24-1094342019839124","volume-title":"Getting Started with Hazelcast","author":"Johns M","year":"2015"},{"key":"bibr25-1094342019839124","volume-title":"8th Workshop in many-task computing on clouds, grids, and supercomputers (MTAGS) collocated with SC 2015","author":"Kleese van Dam K","year":"2015"},{"key":"bibr26-1094342019839124","author":"Koch D","year":"2016","journal-title":"AGU Fall Meeting Abstracts"},{"key":"bibr47-1094342019839124","unstructured":"Langford J, Larochelle H. (2017) Reproduciblity in Machine Learning. In: 34th International Conference on Machine Learning (ICML 2017). Reproduciblity Workshop, Sydney, Australia, 6\u201311 August 2017."},{"key":"bibr27-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3125"},{"key":"bibr28-1094342019839124","unstructured":"Merzky A, Santcroos M, Turilli M, et al. (2016) Executing dynamic and heterogeneous workloads on super computers. arXiv preprint:1512.08194v2."},{"key":"bibr29-1094342019839124","doi-asserted-by":"publisher","DOI":"10.2200\/S00528ED1V01Y201308WBE007"},{"key":"bibr30-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/3097766.3097773"},{"key":"bibr31-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2017.101"},{"key":"bibr32-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1126\/science.1213847"},{"key":"bibr33-1094342019839124","volume-title":"Digital Java EE 7 Web Application Development","author":"Pilgrim P","year":"2015"},{"key":"bibr34-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1006\/jcph.1995.1039"},{"key":"bibr35-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/NYSDS.2018.8538951"},{"key":"bibr36-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/NYSDS.2017.8085043"},{"key":"bibr37-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003285"},{"key":"bibr38-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840839"},{"key":"bibr39-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1126\/science.aah6168"},{"key":"bibr40-1094342019839124","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.TN.1297"},{"key":"bibr41-1094342019839124","unstructured":"Turilli M, Merzky A, Jha S (2016) Designing workflow systems using building blocks. arXiv preprint arXiv:1609.03484."},{"key":"bibr49-1094342019839124","doi-asserted-by":"crossref","unstructured":"Valiev M, Bylaska EJ, Govind N, et al. (2010) NWChem: a comprehensive and scalable open-source solution for large scale molecular simulations. Computer Physics Communications 181(9): 1477\u20131489.","DOI":"10.1016\/j.cpc.2010.04.018"},{"key":"bibr42-1094342019839124","doi-asserted-by":"publisher","DOI":"10.2172\/1473756"},{"key":"bibr43-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1145\/2442776.2442781"},{"key":"bibr44-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498277"},{"key":"bibr45-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/NYSDS.2016.7747811"},{"key":"bibr46-1094342019839124","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2016.38"},{"key":"bibr48-1094342019839124","doi-asserted-by":"publisher","DOI":"10.5220\/0006646803330340"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342019839124","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342019839124","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342019839124","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T08:05:56Z","timestamp":1740816356000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342019839124"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,8]]},"references-count":49,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.1177\/1094342019839124"],"URL":"https:\/\/doi.org\/10.1177\/1094342019839124","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4,8]]}}}