{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,26]],"date-time":"2026-07-26T04:36:25Z","timestamp":1785040585033,"version":"3.55.0"},"reference-count":204,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T00:00:00Z","timestamp":1736899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>The recognized credibility crisis in scientific research has led to an increasing focus on reproducibility studies, particularly in computer science. Existing studies predominantly examine specific technological aspects of reproducibility but neglect the critical interplay between authors and publishers in enabling reproducible computational scientific research.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A systematic review was conducted following the PRISMA Literature Review methodology, complemented by a Journals Survey. This approach enabled a comprehensive analysis of reproducibility policies and software\/data management practices in scientific computing journals.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The survey revealed significant variability in reproducibility policies and practices across computer science journals. Many gaps and challenges were identified, including inconsistencies in policy enforcement, lack of standardized tools, and insufficient recognition of software as a research artifact. The analysis highlighted the potential of Reproducibility as a Service (RaaS) as an innovative solution to address these challenges.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>This study underscores the need for improved standardization and implementation of reproducibility policies. Strategies to enhance reproducibility include fostering collaboration among authors, publishers, and technology providers, as well as recognizing software as a critical research output. The findings aim to guide stakeholders in bridging the current gaps and advancing the reproducibility of computational scientific articles.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fcomp.2024.1491823","type":"journal-article","created":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T06:16:10Z","timestamp":1736921770000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Reproducible research policies and software\/data management in scientific computing journals: a survey, discussion, and perspectives"],"prefix":"10.3389","volume":"6","author":[{"given":"Jose Armando","family":"Hernandez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miguel","family":"Colom","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,1,15]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"965","DOI":"10.1038\/s41562-021-01190-w","volume":"5","year":"","journal-title":"Nat. Hum. Behav"},{"key":"B2","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/MCSE.2021.3059437","article-title":"Cloud-native repositories for big scientific data","volume":"23","author":"Abernathey","year":"2021","journal-title":"Comput. Sci. Eng"},{"key":"B3","doi-asserted-by":"publisher","first-page":"100576","DOI":"10.1016\/j.softx.2020.100576","article-title":"YAWL: an open source business process management system from science for science","volume":"12","author":"Adams","year":"2020","journal-title":"SoftwareX"},{"key":"B4","doi-asserted-by":"publisher","first-page":"W537","DOI":"10.1093\/nar\/gky379","article-title":"The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update","volume":"46","author":"Afgan","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1021","DOI":"10.1038\/s41562-023-01637-2","article-title":"The future of academic publishing","volume":"7","author":"Ahmed","year":"2023","journal-title":"Nat. Hum. Behav"},{"key":"B6","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1109\/TransAI54797.2022.00008","article-title":"\u201cMeasuring reproduciblity of machine learning methods for medical diagnosis,\u201d","volume-title":"2022 Fourth International Conference on Transdisciplinary AI (TransAI)","author":"Ahmed","year":"2022"},{"key":"B7","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MCSE.2021.3072860","article-title":"Toward long-term and archivable reproducibility","volume":"23","author":"Akhlaghi","year":"2021","journal-title":"Comput. Sci. Eng"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.12691","article-title":"Reproducibility of machine learning: terminology, recommendations and open issues","author":"Albertoni","year":"2023","journal-title":"arXiv [Preprint]"},{"key":"B9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2443416.2443417","article-title":"\u201cMakeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids,\u201d","volume-title":"Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies","author":"Albrecht","year":"2012"},{"key":"B10","doi-asserted-by":"publisher","first-page":"e163","DOI":"10.7717\/peerj-cs.163","article-title":"Towards computational reproducibility: researcher perspectives on the use and sharing of software","volume":"4","author":"AlNoamany","year":"2018","journal-title":"PeerJ Comput. Sci"},{"key":"B11","first-page":"291","article-title":"\u201cSoftware engineering for machine learning: a case study,\u201d","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)","author":"Amershi","year":"2019"},{"key":"B12","doi-asserted-by":"publisher","first-page":"8992","DOI":"10.1109\/ACCESS.2020.2964587","article-title":"Resources for reproducibility of experiments in empirical software engineering: topics derived from a secondary study","volume":"8","author":"Anchundia","year":"2020","journal-title":"IEEE Access"},{"key":"B13","doi-asserted-by":"publisher","first-page":"295","DOI":"10.12688\/f1000research.23224.1","article-title":"An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action","volume":"9","author":"Anzt","year":"2020","journal-title":"F1000Res"},{"key":"B14","year":"2009","journal-title":"Above the Clouds: A Berkeley View of Cloud Computing."},{"key":"B15","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1145\/3552490.3552511","article-title":"Artifacts availability & reproducibility (VLDB 2021 round table)","volume":"51","author":"Athanassoulis","year":"2022","journal-title":"ACM SIGMOD Rec"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2109.10387","article-title":"Toward reusable science with readable code and reproducibility","author":"Bahaidarah","year":"2021","journal-title":"arXiv [Preprint]"},{"key":"B17","doi-asserted-by":"publisher","first-page":"483","DOI":"10.1177\/1094342020938424","article-title":"Reproducibility and variable precision computing","volume":"34","author":"Bailey","year":"2020","journal-title":"Int. J. High Perform. Comput. Appl"},{"key":"B18","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1109\/JPROC.2018.2816618","article-title":"Reflections on the future of research curation and research reproducibility [point of view]","volume":"106","author":"Baillieul","year":"2018","journal-title":"Proc. IEEE"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1038\/nature.2016.20504","article-title":"Why scientists must share their research code","author":"Baker","year":"2016","journal-title":"Nature"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.2172\/1478744","article-title":"Workshop report on basic research needs for scientific machine learning: core technologies for artificial intelligence","author":"Baker","year":"2019","journal-title":"Tech. Rep"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2202.12443","article-title":"Towards an accountable and reproducible federated learning: a factsheets approach","author":"Baracaldo","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B22","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1109\/MC.2022.3177133","article-title":"Defining the role of open source software in research reproducibility","volume":"55","author":"Barba","year":"2022","journal-title":"Computer"},{"key":"B23","doi-asserted-by":"publisher","first-page":"622","DOI":"10.1038\/s41597-022-01710-x","article-title":"Introducing the FAIR principles for research software","volume":"9","author":"Barker","year":"2022","journal-title":"Sci. Data"},{"key":"B24","doi-asserted-by":"publisher","first-page":"giaa121","DOI":"10.1093\/gigascience\/giaa121","article-title":"Unifying package managers, workflow engines, and containers: computational reproducibility with BioNix","volume":"9","author":"Bed\u0151","year":"2024","journal-title":"GigaScience"},{"key":"B25","doi-asserted-by":"publisher","first-page":"69","DOI":"10.3389\/fninf.2017.00069","article-title":"Re-run, repeat, reproduce, reuse, replicate: transforming code into scientific contributions","volume":"11","author":"Benureau","year":"2018","journal-title":"Front. Neuroinform"},{"key":"B26","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1109\/MRA.2017.2722918","article-title":"A new kind of article for reproducible research in intelligent robotics [from the field]","volume":"24","author":"Bonsignorio","year":"2017","journal-title":"IEEE Robot. Autom. Mag"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4558704","author":"Bosman","year":"2021","journal-title":"OA Diamond Journals Study. Part 1: Findings"},{"key":"B28","doi-asserted-by":"publisher","first-page":"3447","DOI":"10.1158\/1078-0432.CCR-18-0227","article-title":"A survey on data reproducibility and the effect of publication process on the ethical reporting of laboratory research","volume":"24","author":"Boulbes","year":"2018","journal-title":"Clin. Cancer Res"},{"key":"B29","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1038\/s43586-023-00218-x","article-title":"Improving instrument reproducibility with open source hardware","volume":"3","author":"Bowman","year":"2023","journal-title":"Nat. Rev. Methods Primers"},{"key":"B30","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1109\/BigData.2017.8258038","article-title":"\u201cThe ML test score: a rubric for ml production readiness and technical debt reduction,\u201d","volume-title":"2017 IEEE International Conference on Big Data (Big Data)","author":"Breck","year":"2017"},{"key":"B31","doi-asserted-by":"publisher","first-page":"854","DOI":"10.1016\/j.future.2017.12.029","article-title":"Computing environments for reproducibility: capturing the \u201cwhole tale\"","volume":"94","author":"Brinckman","year":"2019","journal-title":"Future Gener. Comput. Syst"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2004.07213","article-title":"Toward trustworthy AI development: mechanisms for supporting verifiable claims","author":"Brundage","year":"2020","journal-title":"arXiv [Preprint]"},{"key":"B33","first-page":"519","article-title":"\u201cThe state of reproducible research in computer science,\u201d","volume-title":"17th International Conference on Information Technology-New Generations (ITNG 2020)","author":"Cacho","year":"2020"},{"key":"B34","first-page":"19","article-title":"\u201cThe role of containers in reproducibility,\u201d","volume-title":"2020 2nd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)","author":"Canon","year":"2020"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.3233\/APC200107","article-title":"\u201cToward enabling reproducibility for data-intensive research using the whole tale platform,\u201d","author":"Chard","year":"2020","journal-title":"Advances in Parallel Computing"},{"key":"B36","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/3322790.3330594","article-title":"\u201cImplementing computational reproducibility in the whole tale environment,\u201d","volume-title":"Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Chard","year":"2019"},{"key":"B37","unstructured":"\u201cReprozip: using provenance to support computational reproducibility,\u201d\n          \n          \n            \n              Chirigati\n              F.\n            \n            \n              Shasha\n              D.\n            \n            \n              Freire\n              J.\n            \n          \n          5th USENIX Conference on Theory and Practice of Provenance (TaPP'13)\n          \n          2013"},{"key":"B38","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1016\/j.future.2017.01.012","article-title":"Scientific workflows for computational reproducibility in the life sciences: status, challenges and opportunities","volume":"75","author":"Cohen-Boulakia","year":"2017","journal-title":"Future Gener. Comput. Syst"},{"key":"B39","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1145\/2812803","article-title":"Repeatability in computer systems research","volume":"59","author":"Collberg","year":"2016","journal-title":"Commun. ACM"},{"key":"B40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/NTMS.2015.7266500","article-title":"\u201cIPOL: a new journal for fully reproducible research; analysis of four years development,\u201d","volume-title":"2015 7th International Conference on New Technologies, Mobility and Security (NTMS)","author":"Colom","year":"2015"},{"key":"B41","volume-title":"Reproducibility and Replicability in Science","year":"2019"},{"key":"B42","doi-asserted-by":"publisher","first-page":"187","DOI":"10.25080\/Majora-7b98e3ed-01d","author":"Congo","year":"2015","journal-title":"Building a Cloud Service for Reproducible Simulation Management"},{"key":"B43","doi-asserted-by":"publisher","first-page":"180259","DOI":"10.1038\/sdata.2018.259","article-title":"A data citation roadmap for scientific publishers","volume":"5","author":"Cousijn","year":"2018","journal-title":"Sci. Data"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1504.01310","article-title":"Reproducibility as a technical specification","author":"Crick","year":"2015","journal-title":"arXiv [Preprint]"},{"key":"B45","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1145\/3486897","article-title":"Methods included: standardizing computational reuse and portability with the common workflow language","volume":"65","author":"Crusoe","year":"2022","journal-title":"Commun. ACM"},{"key":"B46","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1186\/s12916-020-01566-1","article-title":"Checklists to detect potential predatory biomedical journals: a systematic review","volume":"18","author":"Cukier","year":"2020","journal-title":"BMC Med"},{"key":"B47","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1109\/WORKS54523.2021.00016","article-title":"\u201cA community roadmap for scientific workflows research and development,\u201d","volume-title":"2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS)","author":"Da Silva","year":"2021"},{"key":"B48","article-title":"\u201cInfrastructure-as-code for scientific computing environments,\u201d","author":"Daniel Adorno Gomes","year":"2019","journal-title":"CENTRIC 2019: The Twelfth International Conference on Advances in Human-Oriented and Personalized Mechanisms."},{"key":"B49","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/s10915-023-02193-7","article-title":"Enhancing reproducibility of research papers in SISC, JSC and JCP","volume":"95","author":"De Sterck","year":"2023","journal-title":"J. Sci. Comput"},{"key":"B50","first-page":"11","author":"Deelman","year":"2004"},{"key":"B51","doi-asserted-by":"publisher","first-page":"835","DOI":"10.1109\/COMSNETS56262.2023.10041378","article-title":"\u201cExperimental research reproducibility and experiment workflow management,\u201d","author":"Demchenko","year":"2023","journal-title":"2023 15th International Conference on COMmunication Systems"},{"key":"B52","doi-asserted-by":"publisher","author":"Di Cosmo","year":"2022","DOI":"10.17605\/OSF.IO\/KDE56"},{"key":"B53","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1186\/s13104-021-05875-3","article-title":"Reproducibility and research integrity: the role of scientists and institutions","volume":"14","author":"Diaba-Nuhoho","year":"2021","journal-title":"BMC Res. Notes"},{"key":"B54","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1002\/leap.1258","article-title":"The future of academic publishing: revolution or evolution revisited","volume":"32","author":"Dodds","year":"2019","journal-title":"Learn. Publ"},{"key":"B55","doi-asserted-by":"publisher","first-page":"198","DOI":"10.3389\/fbioe.2018.00198","article-title":"GitHub statistics as a measure of the impact of open-source bioinformatics software","volume":"6","author":"Dozmorov","year":"2018","journal-title":"Front. Bioeng. Biotechnol"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.09015","article-title":"Software publications with rich metadata: state of the art, automated workflows and HERMES concept","author":"Druskat","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B57","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1089\/ees.2016.0223","article-title":"Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and hypercompetition","volume":"34","author":"Edwards","year":"2017","journal-title":"Environ. Eng. Sci"},{"key":"B58","doi-asserted-by":"publisher","first-page":"104753","DOI":"10.1016\/j.envsoft.2020.104753","article-title":"A taxonomy for reproducible and replicable research in environmental modelling","volume":"134","author":"Essawy","year":"2020","journal-title":"Environ. Modell. Softw"},{"key":"B59","doi-asserted-by":"publisher","first-page":"17","DOI":"10.3390\/mti6020017","article-title":"Reproducibility: a researcher-centered definition","volume":"6","author":"Feger","year":"2022","journal-title":"Multimodal Technol. Interact"},{"key":"B60","unstructured":"Forde\n              J.\n            \n            \n              Head\n              T.\n            \n            \n              Holdgraf\n              C.\n            \n            \n              Panda\n              Y.\n            \n            \n              Nalvarete\n              G.\n            \n            \n              Ragan-Kelley\n              B.\n            \n          \n          Reproducible Research Environments With Repo2Docker\n          \n          2018"},{"key":"B61","first-page":"100","article-title":"\u201cReproducibility requires consolidated artifacts,\u201d","volume-title":"2023 IEEE\/ACM 2nd International Conference on AI Engineering","author":"Fostiropoulos","year":"2023"},{"key":"B62","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1145\/3472749.3474734","article-title":"\u201cMarcelle: composing interactive machine learning workflows and interfaces,\u201d","volume-title":"The 34th Annual ACM Symposium on User Interface Software and Technology","author":"Fran\u00e7oise","year":"2021"},{"key":"B63","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1145\/2213836.2213908","article-title":"\u201cComputational reproducibility: state-of-the-art, challenges, and database research opportunities,\u201d","volume-title":"Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data","author":"Freire","year":"2012"},{"key":"B64","doi-asserted-by":"publisher","first-page":"4988","DOI":"10.1109\/JSTARS.2020.3019418","article-title":"A badging system for reproducibility and replicability in remote sensing research","volume":"13","author":"Frery","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens"},{"key":"B65","doi-asserted-by":"publisher","first-page":"790","DOI":"10.1002\/spe.3169","article-title":"Adopting microservices and DevOps in the cyber-physical systems domain: a rapid review and case study. software: practice and experience","volume":"53","author":"Fritzsch","year":"2023","journal-title":"Softw. Pract. Exp"},{"key":"B66","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1145\/3589806.3600033","article-title":"\u201cWe need more reproducibility content across the computer science curriculum,\u201d","volume-title":"Proceedings of the 2023 ACM Conference on Reproducibility and Replicability","author":"Fund","year":"2023"},{"key":"B67","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/EMC2.2018.00008","article-title":"\u201cInvited talk abstract: introducing ReQuEST: an open platform for reproducible and quality-efficient systems-ML tournaments,\u201d","volume-title":"2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2)","author":"Fursin","year":"2018"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2011.01149","article-title":"Collective knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs","author":"Fursin","year":"","journal-title":"arXiv [Preprint]"},{"key":"B69","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2006.07161","article-title":"The collective knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps","author":"Fursin","year":"","journal-title":"arXiv [Preprint]"},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.4005773","article-title":"Enabling reproducible ML and Systems research: the good, the bad, and the ugly","author":"Fursin","year":"","journal-title":"Zenodo"},{"key":"B71","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1155\/2014\/797348","article-title":"Collective mind: towards practical and collaborative auto-tuning","volume":"22","author":"Fursin","year":"2014","journal-title":"Sci. Program"},{"key":"B72","doi-asserted-by":"crossref","DOI":"10.1201\/9780429031854","volume-title":"Reproducible research with R and RStudio. The R series","author":"Gandrud","year":"2020"},{"key":"B73","doi-asserted-by":"publisher","DOI":"10.3390\/books978-3-03928-315-6-11","article-title":"\u201cIncentives, rewards, and recognition - what really motivates a researcher?,\u201d","author":"Ghimpau","year":"2019","journal-title":"Judging Research (MDPI)."},{"key":"B74","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/3456287.3465478","article-title":"\u201cExperiences with reproducibility: case studies from scientific workflows,\u201d","volume-title":"Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Ghoshal","year":"2020"},{"key":"B75","volume-title":"Practical MLOps: operationalizing machine learning models","author":"Gift","year":"2021"},{"key":"B76","doi-asserted-by":"publisher","first-page":"20221113","DOI":"10.1098\/rspb.2022.1113","article-title":"Why don't we share data and code? Perceived barriers and benefits to public archiving practices","volume":"289","author":"Gomes","year":"2022","journal-title":"Proc. R. Soc. B Biol. Sci"},{"key":"B77","doi-asserted-by":"publisher","first-page":"107318","DOI":"10.1016\/j.infsof.2023.107318","article-title":"Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories","volume":"164","author":"Gonzalez-Barahona","year":"2023","journal-title":"Inf. Softw. Technol"},{"key":"B78","author":"Gonzalo Rivero","year":"2020","journal-title":"Best Coding Practices to Ensure Reproducibility."},{"key":"B79","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1609\/aimag.v41i3.5318","article-title":"The reproducibility crisis is real","volume":"41","author":"Gundersen","year":"2020","journal-title":"AI Mag"},{"key":"B80","doi-asserted-by":"publisher","first-page":"20200210","DOI":"10.1098\/rsta.2020.0210","article-title":"The fundamental principles of reproducibility","volume":"379","author":"Gundersen","year":"2021","journal-title":"Philos. Trans. R. Soc. A Math. Phys. Eng. Sci"},{"key":"B81","doi-asserted-by":"publisher","first-page":"11503","DOI":"10.1609\/aaai.v32i1.11503","article-title":"State of the art: reproducibility in artificial intelligence","volume":"32","author":"Gundersen","year":"2018","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B82","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.future.2021.06.014","article-title":"Do machine learning platforms provide out-of-the-box reproducibility?","volume":"126","author":"Gundersen","year":"2022","journal-title":"Future Gener. Comput. Syst"},{"key":"B83","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2202.07435","article-title":"State of AI ethics report","author":"Gupta","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B84","doi-asserted-by":"publisher","first-page":"E14","DOI":"10.1038\/s41586-020-2766-y","article-title":"Transparency and reproducibility in artificial intelligence","volume":"586","author":"Haibe-Kains","year":"2020","journal-title":"Nature"},{"key":"B85","first-page":"156","article-title":"\u201cHow to open science: a principle and reproducibility review of the learning analytics and knowledge conference,\u201d","volume-title":"LAK23: 13th International Learning Analytics and Knowledge Conference","author":"Haim","year":""},{"key":"B86","first-page":"74","article-title":"\u201cHow to open science: promoting principles and reproducibility practices within the artificial intelligence in education community,\u201d","volume-title":"Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Vol. 1831","author":"Haim","year":""},{"key":"B87","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1086\/693875","article-title":"Communism and the incentive to share in science","volume":"84","author":"Heesen","year":"2017","journal-title":"Philos. Sci"},{"key":"B88","doi-asserted-by":"publisher","DOI":"10.2172\/1481626","article-title":"Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences","author":"Heroux","year":"2018","journal-title":"Tech. Rep."},{"key":"B89","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.02544","article-title":"The race to the vulnerable: measuring the log4j shell incident","author":"Hiesgen","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B90","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1145\/3366424.3383417","article-title":"\u201cExpanding the scope of reproducibility research through data analysis replications,\u201d","volume-title":"Companion Proceedings of the Web Conference 2020","author":"Hofman","year":"2020"},{"key":"B91","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1109\/MCSE.2012.62","article-title":"Virtual appliances, cloud computing, and reproducible research","volume":"14","author":"Howe","year":"2012","journal-title":"Comput. Sci. Eng"},{"key":"B92","doi-asserted-by":"publisher","first-page":"5175","DOI":"10.1038\/s41467-020-18964-x","article-title":"Evolution of DNA replication origin specification and gene silencing mechanisms","volume":"11","author":"Hu","year":"2020","journal-title":"Nat. Commun"},{"key":"B93","first-page":"6852","article-title":"\u201cHistory of technology,\u201d","volume-title":"International Encyclopedia of the Social","author":"Hughes","year":"2001"},{"key":"B94","doi-asserted-by":"publisher","first-page":"W729","DOI":"10.1093\/nar\/gkl320","article-title":"Taverna: a tool for building and running workflows of services","volume":"34","author":"Hull","year":"2006","journal-title":"Nucleic Acids Res"},{"key":"B95","unstructured":"\u201cA literature review on reproducibility studies in computer science,\u201d\n          \n          \n            \n              Hummel\n              T.\n            \n            \n              Manner\n              J.\n            \n          \n          CEUR\n          Proceedings of the 16th ZEUS Workshop on Services and Their Composition (ZEUS 2024)\n          \n          2024"},{"key":"B96","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1126\/science.359.6377.725","article-title":"Artificial intelligence faces reproducibility crisis","volume":"359","author":"Hutson","year":"2018","journal-title":"Science"},{"key":"B97","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/978-3-319-16462-5_32","article-title":"\u201cProvStore: a public provenance repository,\u201d","volume-title":"Provenance and Annotation of Data and Processes, Vol. 8628","author":"Huynh","year":"2015"},{"key":"B98","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/978-3-031-17105-5_6","article-title":"\u201cDocumenting the creation, manipulation and evaluation of links for reuse and reproducibility,\u201d","volume-title":"Knowledge Engineering and Knowledge Management, Vol. 13514","author":"Idrissou","year":"2022"},{"key":"B99","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1145\/3519935.3519973","article-title":"\u201cReproducibility in learning,\u201d","volume-title":"Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing","author":"Impagliazzo","year":"2022"},{"key":"B100","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3186266","article-title":"Reproducibility in scientific computing","volume":"63","author":"Ivie","year":"2018","journal-title":"ACM Comput. Surv"},{"key":"B101","doi-asserted-by":"crossref","first-page":"066","DOI":"10.1109\/EIT48999.2020.9208338","article-title":"\u201cImproving numerical reproducibility of scientific software in parallel systems,\u201d","volume-title":"2020 IEEE International Conference on Electro Information Technology (EIT)","author":"Jalal Apostal","year":"2020"},{"key":"B102","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.00077","article-title":"SciOps: achieving productivity and reliability in data-intensive research","author":"Johnson","year":"2024","journal-title":"arXiv [Preprint]"},{"key":"B103","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1902.03383","article-title":"Cloud programming simplified: a Berkeley view on serverless computing","author":"Jonas","year":"2019","journal-title":"arXiv [Preprint]"},{"key":"B104","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2207.07048","article-title":"Leakage and the reproducibility crisis in ML-based science","author":"Kapoor","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B105","doi-asserted-by":"publisher","first-page":"799","DOI":"10.1038\/s42256-023-00652-2","article-title":"Federated benchmarking of medical artificial intelligence with MedPerf","volume":"5","author":"Karargyris","year":"2023","journal-title":"Nat. Mach. Intell"},{"key":"B106","doi-asserted-by":"publisher","first-page":"e1010615","DOI":"10.1371\/journal.pcbi.1010615","article-title":"Reproducibility efforts as a teaching tool: a pilot study","volume":"18","author":"Karathanasis","year":"2022","journal-title":"PLoS Comput. Biol"},{"key":"B107","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1109\/BRAINS49436.2020.9223307","article-title":"\u201cAI pedigree verification platform using blockchain,\u201d","author":"Kawamoto","year":"2020","journal-title":"2020 2nd Conference on Blockchain Research"},{"key":"B108","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2305.17033","article-title":"The brain tumor segmentation (BraTS) challenge 2023: focus on pediatrics (CBTN-CONNECT-DIPGR-ASNR1229 MICCAI BraTS-PEDs)","author":"Kazerooni","year":"2023","journal-title":"arXiv [Preprint]"},{"key":"B109","doi-asserted-by":"publisher","first-page":"e2309378120","DOI":"10.1073\/pnas.2309378120","article-title":"A network-based normalized impact measure reveals successful periods of scientific discovery across disciplines","volume":"120","author":"Ke","year":"2023","journal-title":"Proc. Nat. Acad. Sci"},{"key":"B110","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1201\/9781351036863-5","article-title":"\u201cChameleon: a scalable production testbed for computer science research,\u201d","volume-title":"Contemporary High Performance Computing","author":"Keahey","year":"2019"},{"key":"B111","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.12322","article-title":"MLDev: data science experiment automation and reproducibility software","author":"Khritankov","year":"2021","journal-title":"arXiv [Preprint]"},{"key":"B112","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1007\/s10664-019-09747-0","article-title":"Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment. Empirical","volume":"25","author":"Kitchenham","year":"2020","journal-title":"Softw. Eng"},{"key":"B113","volume-title":"The practice of reproducible research: case studies and lessons from the data-intensive sciences","author":"Kitzes","year":"2018"},{"key":"B114","doi-asserted-by":"publisher","first-page":"giaa068","DOI":"10.1093\/gigascience\/giaa068","article-title":"Watchdog 2.0: new developments for reusability, reproducibility, and workflow execution","volume":"9","author":"Kluge","year":"2020","journal-title":"GigaScience"},{"key":"B115","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/s41073-020-00095-y","article-title":"Publishing computational research - a review of infrastructures for reproducible and transparent scholarly communication","volume":"5","author":"Konkol","year":"2020","journal-title":"Res. Integr. Peer Rev"},{"key":"B116","first-page":"109","article-title":"Shiwa workflow interoperability solutions for neuroimaging data analysis","volume":"175","author":"Korkhov","year":"2012","journal-title":"Stud. Health Technol. Inform"},{"key":"B117","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1109\/MCSE.2012.38","article-title":"Reproducible research for scientific computing: tools and strategies for changing the culture","volume":"14","author":"LeVeque","year":"2012","journal-title":"Comput. Sci. Eng"},{"key":"B118","volume-title":"From policy to practice: How journal-based data policies encourage scientists' adoption of reproducible research practices","author":"Lewis","year":"2023"},{"key":"B119","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2212.13534","article-title":"Building a culture of reproducibility in academic research","author":"Lin","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B120","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/978-3-030-45442-5_6","article-title":"\u201cReproducibility is a process, not an achievement: the replicability of IR reproducibility experiments,\u201d","volume-title":"Advances in Information Retrieval, Volume 12036","author":"Lin","year":"2020"},{"key":"B121","doi-asserted-by":"publisher","first-page":"24","DOI":"10.21037\/atm.2017.06.24","article-title":"Scientist impact factor (SIF): a new metric for improving scientists' evaluation?","volume":"5","author":"Lippi","year":"2017","journal-title":"Ann. Transl. Med"},{"key":"B122","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1007\/978-3-030-76423-4_2","article-title":"\u201cReproducibility: evaluating the evaluations,\u201d","volume-title":"Reproducible Research in Pattern Recognition, Vol. 12636","author":"Lopresti","year":"2021"},{"key":"B123","doi-asserted-by":"publisher","first-page":"12792","DOI":"10.1609\/aaai.v36i11.21558","article-title":"Reproducibility as a mechanism for teaching fairness, accountability, confidentiality, and transparency in artificial intelligence","volume":"36","author":"Lucic","year":"2022","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B124","doi-asserted-by":"publisher","DOI":"10.1117\/3.2317707","author":"Mack","year":"2018","journal-title":"How to Write a Good Scientific Paper"},{"key":"B125","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1186\/s13104-022-06030-2","article-title":"Improving the reproducibility and integrity of research: what can different stakeholders contribute?","volume":"15","author":"Macleod","year":"2022","journal-title":"BMC Res. Notes"},{"key":"B126","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3456287.3465479","article-title":"\u201cArtifact description\/artifact evaluation: a reproducibility bane or a boon,\u201d","volume-title":"Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Malik","year":"2020"},{"key":"B127","doi-asserted-by":"crossref","first-page":"2313","DOI":"10.1109\/BigData52589.2021.9671588","article-title":"\u201cA survey study of success factors in data science projects,\u201d","volume-title":"2021 IEEE International Conference on Big Data (Big Data)","author":"Martinez","year":"2021"},{"key":"B128","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3528231.3528359","article-title":"\u201cBeyond the badge: reproducibility engineering as a lifetime skill,\u201d","volume-title":"Proceedings of the 4th International Workshop on Software Engineering Education for the Next Generation","author":"Mauerer","year":"2022"},{"key":"B129","first-page":"147","article-title":"\u201cA model-driven approach for systematic reproducibility and replicability of data science projects,\u201d","volume-title":"Advanced Information Systems Engineering, Vol","author":"Melchor","year":"2022"},{"key":"B130","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1016\/j.procs.2017.05.116","article-title":"Facilitating the reproducibility of scientific workflows with execution environment specifications","volume":"108","author":"Meng","year":"2017","journal-title":"Procedia Comput. Sci"},{"key":"B131","doi-asserted-by":"publisher","first-page":"5868","DOI":"10.1021\/acs.jcim.0c01389","article-title":"Editorial: Method and data sharing and reproducibility of scientific results","volume":"60","author":"Merz","year":"2020","journal-title":"J. Chem. Inf. Model"},{"key":"B132","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1145\/3589806.3600040","article-title":"\u201cTowards evidence-based software quality practices for reproducibility: preliminary results and research directions,\u201d","volume-title":"Proceedings of the 2023 ACM Conference on Reproducibility and Replicability","author":"Milewicz","year":"2023"},{"key":"B133","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s43586-023-00236-9","article-title":"Containers for computational reproducibility","volume":"3","author":"Moreau","year":"2023","journal-title":"Nat. Rev. Methods Primers"},{"key":"B134","doi-asserted-by":"publisher","first-page":"e2100769118","DOI":"10.1073\/pnas.2100769118","article-title":"A better approach for dealing with reproducibility and replicability in science","volume":"118","author":"Nichols","year":"2021","journal-title":"Proc. Natl. Acad. Sci. U S A"},{"key":"B135","doi-asserted-by":"publisher","DOI":"10.21203\/rs.3.rs-2267847\/v1","article-title":"A literature review of methods for assessment of reproducibility in science","author":"Nordling","year":"2022","journal-title":"Res. Sq"},{"key":"B136","unstructured":"State of the Octoverse 2024\n          \n          2024"},{"key":"B137","first-page":"220","article-title":"\u201cReproducibility of computational experiments on kubernetes-managed container clouds with hyperflow,\u201d","volume-title":"Computational Science","author":"Orzechowski","year":"2020"},{"key":"B138","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/MC.2022.3176988","article-title":"Research reproducibility","volume":"55","author":"Parashar","year":"2022","journal-title":"Computer"},{"key":"B139","doi-asserted-by":"publisher","first-page":"76084","DOI":"10.23978\/inf.76084","article-title":"Supporting FAIR data: categorization of research data as a tool in data management","volume":"37","author":"Parland-von Essen","year":"2018","journal-title":"Informaatiotutkimus"},{"key":"B140","doi-asserted-by":"publisher","first-page":"52","DOI":"10.5334\/dsj-2019-052","article-title":"The history and future of data citation in practice","volume":"18","author":"Parsons","year":"2019","journal-title":"Data Sci. J"},{"key":"B141","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/978-3-031-23298-5_1","article-title":"\u201cTowards a metadata management system for provenance, reproducibility and accountability in federated machine learning,\u201d","volume-title":"Advances in Service-Oriented and Cloud Computing, Vol. 1617","author":"Peregrina","year":"2022"},{"key":"B142","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1126\/science.aaw2825","article-title":"Certify reproducibility with confidential data","volume":"365","author":"P\u00e9rignon","year":"2019","journal-title":"Science"},{"key":"B143","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1109\/MSR.2019.00077","article-title":"\u201cA large-scale study about quality and reproducibility of jupyter notebooks,\u201d","volume-title":"2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR)","author":"Pimentel","year":"2019"},{"key":"B144","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s10664-021-09961-9","article-title":"Understanding and improving the quality and reproducibility of Jupyter notebooks","volume":"26","author":"Pimentel","year":"2021","journal-title":"Empir. Softw. Eng"},{"key":"B145","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2003.12206","article-title":"Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program)","author":"Pineau","year":"2020","journal-title":"arXiv [Preprint]"},{"key":"B146","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1109\/MCSE.2021.3096678","article-title":"Reproducibility practice in high-performance computing: community survey results","volume":"23","author":"Plale","year":"2021","journal-title":"Comput. Sci. Eng"},{"key":"B147","doi-asserted-by":"publisher","first-page":"e27725","DOI":"10.7554\/eLife.27725","article-title":"The readability of scientific texts is decreasing over time","volume":"6","author":"Plav\u00e9n-Sigray","year":"2017","journal-title":"Elife"},{"key":"B148","doi-asserted-by":"publisher","first-page":"76","DOI":"10.3389\/fninf.2017.00076","article-title":"Reproducibility vs. replicability: a brief history of a confused terminology","volume":"11","author":"Plesser","year":"2018","journal-title":"Front. Neuroinform"},{"key":"B149","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.neuron.2018.11.030","article-title":"The costs of reproducibility","volume":"101","author":"Poldrack","year":"2019","journal-title":"Neuron"},{"key":"B150","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1177\/1094342019839124","article-title":"Computational reproducibility of scientific workflows at extreme scales","volume":"33","author":"Pouchard","year":"2019","journal-title":"Int. J. High Perform. Comput. Appl"},{"key":"B151","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26050-7_277-1","author":"Prabhu","year":"2020","journal-title":"Reproducible Workflow"},{"key":"B152","author":"Pradal","year":"2019","journal-title":"OpenAlea."},{"key":"B153","article-title":"Ml reproducibility systems: status and research agenda","year":"2021","journal-title":"J. Syst. Res."},{"key":"B154","article-title":"\u201cDelivering cloud native infrastructure as code - pulumi,\u201d","year":"2019","journal-title":"White Paper"},{"key":"B155","doi-asserted-by":"publisher","first-page":"100015","DOI":"10.1016\/j.bcra.2021.100015","article-title":"Verifiable badging system for scientific data reproducibility","volume":"2","author":"Radha","year":"2021","journal-title":"Blockchain Res. Appl"},{"key":"B156","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1909.06674","article-title":"A step toward quantifying independently reproducible machine learning research","author":"Raff","year":"2019","journal-title":"arXiv [Preprint]"},{"key":"B157","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2012.09932","article-title":"Research reproducibility as a survival analysis","author":"Raff","year":"2020","journal-title":"arXiv [Preprint]"},{"key":"B158","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2204.04372","article-title":"A siren song of open source reproducibility","author":"Raff","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B159","doi-asserted-by":"publisher","first-page":"29207","DOI":"10.1109\/ACCESS.2022.3158675","article-title":"Reproducibility in computing research: an empirical study","volume":"10","author":"Raghupathi","year":"2022","journal-title":"IEEE Access"},{"key":"B160","doi-asserted-by":"publisher","first-page":"1595","DOI":"10.1016\/j.jksuci.2021.11.019","article-title":"A review on TinyML: state-of-the-art and prospects","volume":"34","author":"Ray","year":"2022","journal-title":"J. King Saud Univ. Comput. Inf. Sci"},{"key":"B161","doi-asserted-by":"publisher","first-page":"3178","DOI":"10.14778\/3611479.3611517","article-title":"Epistemic parity: reproducibility as an evaluation metric for differential privacy","volume":"16","author":"Rosenblatt","year":"2023","journal-title":"Proc. VLDB Endowment"},{"key":"B162","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1145\/3589806.3600032","article-title":"\u201cKheOps: cost-effective repeatability, reproducibility, and replicability of edge-to-cloud experiments,\u201d","volume-title":"Proceedings of the 2023 ACM Conference on Reproducibility and Replicability","author":"Rosendo","year":"2023"},{"key":"B163","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1007\/978-3-030-23987-9_14","article-title":"\u201cReScience C: a journal for reproducible replications in computational science,\u201d","volume-title":"Reproducible Research in Pattern Recognition, Vol. 11455","author":"Rougier","year":"2019"},{"key":"B164","first-page":"784","author":"Salsabil","year":"2022"},{"key":"B165","doi-asserted-by":"publisher","first-page":"e862","DOI":"10.7717\/peerj-cs.862","article-title":"Current approaches for executing big data science projects-a systematic literature review","volume":"8","author":"Saltz","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"B166","first-page":"201","article-title":"\u201cReproduceMeGit: a visualization tool for analyzing reproducibility of jupyter notebooks,\u201d","volume-title":"Provenance and Annotation of Data and Processes, Vol. 12839","author":"Samuel","year":""},{"key":"B167","doi-asserted-by":"publisher","first-page":"e11140","DOI":"10.7717\/peerj.11140","article-title":"Understanding experiments and research practices for reproducibility: an exploratory study","volume":"9","author":"Samuel","year":"","journal-title":"PeerJ"},{"key":"B168","doi-asserted-by":"publisher","first-page":"e921","DOI":"10.7717\/peerj-cs.921","article-title":"A collaborative semantic-based provenance management platform for reproducibility","volume":"8","author":"Samuel","year":"","journal-title":"PeerJ Comput. Sci"},{"key":"B169","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13326-021-00253-1","article-title":"End-to-End provenance representation for the understandability and reproducibility of scientific experiments using a semantic approach","volume":"13","author":"Samuel","year":"","journal-title":"J. Biomed. Semantics"},{"key":"B170","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1007\/978-3-030-80960-7_17","article-title":"\u201cMachine learning pipelines: provenance, reproducibility and FAIR data principles,\u201d","volume-title":"Provenance and Annotation of Data and Processes, Volume 12839","author":"Samuel","year":"2021"},{"key":"B171","doi-asserted-by":"publisher","first-page":"giad113","DOI":"10.1093\/gigascience\/giad113","article-title":"Computational reproducibility of jupyter notebooks from biomedical publications","volume":"13","author":"Samuel","year":"2023","journal-title":"GigaScience"},{"key":"B172","doi-asserted-by":"publisher","first-page":"e243180","DOI":"10.1155\/2015\/243180","article-title":"Towards reproducibility in scientific workflows: an infrastructure-based approach","volume":"2015","author":"Santana-Perez","year":"2015","journal-title":"Sci. Program"},{"key":"B173","article-title":"On challenges in machine learning model management","author":"Schelter","year":"2015","journal-title":"IEEE Data Eng. Bull."},{"key":"B174","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2210.11831","article-title":"Management of machine learning lifecycle artifacts: a survey","author":"Schlegel","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B175","doi-asserted-by":"publisher","first-page":"526","DOI":"10.1016\/j.procs.2021.01.199","article-title":"A systematic literature review on applying crisp-dm process model","volume":"181","author":"Schr\u00f6er","year":"2021","journal-title":"Procedia Comput. Sci"},{"key":"B176","article-title":"\u201cHidden technical debt in machine learning systems,\u201d","volume-title":"Advances in Neural Information Processing Systems, Vol. 28","author":"Sculley","year":"2015"},{"key":"B177","doi-asserted-by":"publisher","first-page":"e606","DOI":"10.7717\/peerj-cs.606","article-title":"Provenance-and machine learning-based recommendation of parameter values in scientific workflows","volume":"7","author":"Silva Junior","year":"2021","journal-title":"PeerJ Comput. Sci"},{"key":"B178","doi-asserted-by":"publisher","first-page":"e86","DOI":"10.7717\/peerj-cs.86","article-title":"Software citation principles","volume":"2","author":"Smith","year":"2016","journal-title":"PeerJ Comput. Sci"},{"key":"B179","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1186\/s41239-019-0175-9","article-title":"Instilling purpose and value in the implementation of digital badges in higher education","volume":"16","author":"Stefaniak","year":"2019","journal-title":"Int. J. Educ. Technol. High. Educ"},{"key":"B180","doi-asserted-by":"publisher","first-page":"111615","DOI":"10.1016\/j.jss.2023.111615","article-title":"The pipeline for the continuous development of artificial intelligence models-current state of research and practice","volume":"199","author":"Steidl","year":"2023","journal-title":"J. Syst. Softw"},{"key":"B181","doi-asserted-by":"publisher","author":"Stoddart","year":"2016","DOI":"10.1038\/d41586-019-00067-3"},{"key":"B182","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1145\/3391800.3398172","article-title":"\u201cBeyond open data: a model for linking digital artifacts to enable reproducibility of scientific claims,\u201d","volume-title":"Proceedings of the 3rd International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Stodden","year":"2020"},{"key":"B183","author":"Stodden","year":"2012","journal-title":"How Journals Are Adopting Open Data and Code Policies"},{"key":"B184","doi-asserted-by":"publisher","first-page":"e67111","DOI":"10.1371\/journal.pone.0067111","article-title":"Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals","volume":"8","author":"Stodden","year":"2013","journal-title":"PLoS ONE"},{"key":"B185","doi-asserted-by":"publisher","first-page":"e21","DOI":"10.5334\/jors.ay","article-title":"Best practices for computational science: Software infrastructure and environments for reproducible and extensible research","volume":"2","author":"Stodden","year":"2014","journal-title":"J. Open Res. Softw"},{"key":"B186","doi-asserted-by":"publisher","first-page":"2584","DOI":"10.1073\/pnas.1708290115","article-title":"An empirical analysis of journal policy effectiveness for computational reproducibility","volume":"115","author":"Stodden","year":"2018","journal-title":"Proc. Nat. Acad. Sci"},{"key":"B187","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04570","article-title":"Building a reproducible machine learning pipeline","author":"Sugimura","year":"2018","journal-title":"arXiv [Preprint]"},{"key":"B188","doi-asserted-by":"publisher","DOI":"10.5281\/ZENODO.4915998","article-title":"Science europe study on research assessment practices","year":"2020","journal-title":"Zenodo."},{"key":"B189","doi-asserted-by":"publisher","first-page":"W83","DOI":"10.1093\/nar\/gkae410","article-title":"The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update","volume":"52","year":"2024","journal-title":"Nucleic Acids Res"},{"key":"B190","doi-asserted-by":"publisher","year":"2022","journal-title":"The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research","DOI":"10.5281\/ZENODO.3233853"},{"key":"B191","article-title":"Reproducible research","author":"Thompson","year":"2012","journal-title":"CORE Issues Profess. Res. Ethics"},{"key":"B192","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1145\/3391800.3398173","article-title":"\u201cAdvancing computational reproducibility in the dataverse data repository platform,\u201d","volume-title":"Proceedings of the 3rd International Workshop on Practical Reproducible Evaluation of Computer Systems","author":"Trisovic","year":"2020"},{"key":"B193","doi-asserted-by":"publisher","first-page":"100099","DOI":"10.1016\/j.patter.2020.100099","article-title":"Reproducibility starts from you today","volume":"1","author":"Turkyilmaz-van der Velden","year":"2020","journal-title":"Patterns"},{"key":"B194","author":"Vanschoren","year":"2014","journal-title":"Open Science in Machine Learning"},{"key":"B195","doi-asserted-by":"publisher","first-page":"e3208","DOI":"10.7717\/peerj.3208","article-title":"Reproducible and reusable research: are journal data sharing policies meeting the mark?","volume":"5","author":"Vasilevsky","year":"2017","journal-title":"PeerJ"},{"key":"B196","unstructured":"Using computing containers and continuous integration to improve numerical research reproducibility\n          \n          27\n          33\n          \n            \n              Vasyukov\n              A.\n            \n            \n              Petrov\n              I.\n            \n          \n          25681136\n          Int. J. Comput\n          30\n          2018"},{"key":"B197","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1145\/2038642.2038650","article-title":"\u201cRepeatability, reproducibility, and rigor in systems research,\u201d","volume-title":"Proceedings of the ninth ACM international conference on Embedded software","author":"Vitek","year":"2011"},{"key":"B198","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1145\/3377812.3390803","article-title":"\u201cRestoring reproducibility of Jupyter notebooks,\u201d","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Companion Proceedings","author":"Wang","year":"2020"},{"key":"B199","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.25982dcf","article-title":"Trust but verify: how to leverage policies, workflows, and infrastructure to ensure computational reproducibility in publication","author":"Willis","year":"2020","journal-title":"Harvard Data Sci. Rev"},{"key":"B200","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICBC51069.2021.9461139","article-title":"\u201cA blockchain-based approach to provenance and reproducibility in research workflows,\u201d","volume-title":"2021 IEEE International Conference on Blockchain and Cryptocurrency (ICBC)","author":"Wittek","year":"2021"},{"key":"B201","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1016\/j.is.2015.12.004","article-title":"Reproducible experiments on dynamic resource allocation in cloud data centers","volume":"59","author":"Wolke","year":"2016","journal-title":"Inf. Syst"},{"key":"B202","doi-asserted-by":"publisher","first-page":"1543","DOI":"10.1002\/spe.3202","article-title":"Reproducibility as a service","volume":"53","author":"Wonsil","year":"","journal-title":"Softw. Pract. Exp."},{"key":"B203","first-page":"1","article-title":"\u201cIntegrated reproducibility with self-describing machine learning models,\u201d","volume-title":"Proceedings of the 2023 ACM Conference on Reproducibility and Replicability","author":"Wonsil","year":""},{"key":"B204","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2012.01172","article-title":"ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility","author":"Yildiz","year":"2021","journal-title":"arXiv [Preprint]"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2024.1491823\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T06:16:54Z","timestamp":1736921814000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2024.1491823\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,15]]},"references-count":204,"alternative-id":["10.3389\/fcomp.2024.1491823"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2024.1491823","relation":{},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,15]]},"article-number":"1491823"}}