{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T14:39:44Z","timestamp":1775745584448,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":69,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,5,21]],"date-time":"2022-05-21T00:00:00Z","timestamp":1653091200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,5,21]]},"DOI":"10.1145\/3510003.3510163","type":"proceedings-article","created":{"date-parts":[[2022,7,5]],"date-time":"2022-07-05T22:42:59Z","timestamp":1657060979000},"page":"2202-2214","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Towards training reproducible deep learning models"],"prefix":"10.1145","author":[{"given":"Boyuan","family":"Chen","sequence":"first","affiliation":[{"name":"Huawei Canada, Kingston, Canada"}]},{"given":"Mingzhi","family":"Wen","sequence":"additional","affiliation":[{"name":"Huawei Technologies, Shenzhen, China"}]},{"given":"Yong","family":"Shi","sequence":"additional","affiliation":[{"name":"Huawei Technologies, Shenzhen, China"}]},{"given":"Dayi","family":"Lin","sequence":"additional","affiliation":[{"name":"Huawei Canada, Kingston, Canada"}]},{"given":"Gopi Krishnan","family":"Rajbahadur","sequence":"additional","affiliation":[{"name":"Huawei Canada, Kingston, Canada"}]},{"given":"Zhen Ming (Jack)","family":"Jiang","sequence":"additional","affiliation":[{"name":"York University, Toronto, Canada"}]}],"member":"320","published-online":{"date-parts":[[2022,7,5]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2021 (accessed August 2021). An open source platform for the machine learning lifecycle. https:\/\/mlflow.org\/"},{"key":"e_1_3_2_1_2_1","unstructured":"2021 (accessed August 2021). Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment"},{"key":"e_1_3_2_1_3_1","unstructured":"2021 (accessed August 2021). The CIFAR-10 and CIFAR-100 datasets. https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html"},{"key":"e_1_3_2_1_4_1","unstructured":"2021 (accessed August 2021). CUDA Toolkit. https:\/\/developer.nvidia.com\/cuda-toolkit"},{"key":"e_1_3_2_1_5_1","unstructured":"2021 (accessed August 2021). Determined AI Reproducibility. https:\/\/docs.determined.ai\/latest\/topic-guides\/training\/reproducibility.html"},{"key":"e_1_3_2_1_6_1","unstructured":"2021 (accessed August 2021). Determinism in Deep Learning (S9911). https:\/\/developer.download.nvidia.com\/video\/gputechconf\/gtc\/2019\/presentation\/s9911-determinism-in-deep-learning.pdf"},{"key":"e_1_3_2_1_7_1","unstructured":"2021 (accessed August 2021). Inconsistent results when using two styles of import statements - Issue 14672. https:\/\/github.com\/keras-team\/keras\/issues\/14672"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"crossref","unstructured":"2021 (accessed August 2021). Migrate your TensorFlow 1 code to TensorFlow 2. https:\/\/www.tensorflow.org\/guide\/migrate","DOI":"10.1007\/978-1-4842-6373-0_1"},{"key":"e_1_3_2_1_9_1","unstructured":"2021 (accessed August 2021). The Mnist Database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/"},{"key":"e_1_3_2_1_10_1","unstructured":"2021 (accessed August 2021). Most common random seeds. https:\/\/www.kaggle.com\/residentmario\/kernel16e284dcb7"},{"key":"e_1_3_2_1_11_1","unstructured":"2021 (accessed August 2021). Notes from the AI Frontier Insights from Hundreds of Use Cases. https:\/\/www.mckinsey.com\/featured-insights\/artificial-intelligence\/notes-from-the-ai-frontier-applications-and-value-of-deep-learning"},{"key":"e_1_3_2_1_12_1","unstructured":"2021 (accessed August 2021). Open-source Version Control System for Machine Learning Projects. https:\/\/dvc.org\/"},{"key":"e_1_3_2_1_13_1","unstructured":"2021 (accessed August 2021). Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A52021PC0206"},{"key":"e_1_3_2_1_14_1","unstructured":"2021 (accessed August 2021). Reproducibility in Pytorch. https:\/\/pytorch.org\/docs\/stable\/notes\/randomness.html"},{"key":"e_1_3_2_1_15_1","unstructured":"2021 (accessed August 2021). Tensorflow Determinism. https:\/\/github.com\/NVIDIA\/framework-determinism"},{"key":"e_1_3_2_1_16_1","unstructured":"2021 (accessed August 2021). TensorFlow GPU Support. https:\/\/www.tensorflow.org\/install\/source#gpu"},{"key":"e_1_3_2_1_17_1","unstructured":"2021 (accessed August 2021). Tensorflow RFC for determinism. https:\/\/github.com\/tensorflow\/community\/blob\/master\/rfcs\/20210119-determinism.md"},{"key":"e_1_3_2_1_18_1","unstructured":"2021 (accessed August 2021). Testing for Deploying Machine Learning Models. https:\/\/developers.google.com\/machine-learning\/testing-debugging\/pipeline\/deploying"},{"key":"e_1_3_2_1_19_1","unstructured":"2021 (accessed August 2021). The Machine Learning Reproducibility Checklist. https:\/\/www.cs.mcgill.ca\/~jpineau\/ReproducibilityChecklist.pdf"},{"key":"e_1_3_2_1_20_1","unstructured":"2021 (accessed August 2021). Unreproducible results when directly import keras in CPU environment - Issue 14671. https:\/\/github.com\/keras-team\/keras\/issues\/14671"},{"key":"e_1_3_2_1_21_1","unstructured":"2022 (accessed Feb 2022). The replication package. https:\/\/github.com\/nemo9cby\/ICSE2022Rep"},{"key":"e_1_3_2_1_22_1","volume-title":"TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2--4, 2016. USENIX Association, 265--283."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_1_24_1","volume-title":"Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings).","author":"Amodei Dario","year":"2016","unstructured":"Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Han, Patrick LeGresley, Xi-angang Li, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Chong Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2016. Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings)."},{"key":"e_1_3_2_1_25_1","volume-title":"On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects. In 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021","author":"Barrak Amine","year":"2021","unstructured":"Amine Barrak, Ellis E. Eghan, and Bram Adams. 2021. On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects. In 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021, Honolulu, HI, USA, March 9--12, 2021. IEEE, 422--433."},{"key":"e_1_3_2_1_26_1","unstructured":"Miles Brundage Shahar Avin Jasmine Wang Haydn Belfield Gretchen Krueger Gillian K. Hadfield Heidy Khlaaf Jingying Yang Helen Toner Ruth Fong Tegan Maharaj Pang Wei Koh Sara Hooker Jade Leung Andrew Trask Emma Bluemke Jonathan Lebensbold Cullen O'Keefe Mark Koren Theo Ryffel J. B. Rubinovitz Tamay Besiroglu Federica Carugati Jack Clark Peter Eckersley Sarah de Haas Maritza Johnson Ben Laurie Alex Ingerman Igor Krawczuk Amanda Askell Rosario Cammarota Andrew Lohn David Krueger Charlotte Stix Peter Henderson Logan Graham Carina Prunkl Bianca Martin Elizabeth Seger Noa Zilberman Se\u00e1n \u00d3 h\u00c9igeartaigh Frens Kroeger Girish Sastry Rebecca Kagan Adrian Weller Brian Tse Elizabeth Barnes Allan Dafoe Paul Scharre Ariel HerbertVoss Martijn Rasser Shagun Sodhani Carrick Flynn Thomas Krendl Gilbert Lisa Dyer Saif Khan Yoshua Bengio and Markus Anderljung. 2020. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. CoRR abs\/2004.07213 (2020). arXiv:2004.07213 https:\/\/arxiv.org\/abs\/2004.07213"},{"key":"e_1_3_2_1_27_1","volume-title":"cuDNN: Efficient Primitives for Deep Learning. CoRR abs\/1410.0759","author":"Chetlur Sharan","year":"2014","unstructured":"Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient Primitives for Deep Learning. CoRR abs\/1410.0759 (2014). arXiv:1410.0759 http:\/\/arxiv.org\/abs\/1410.0759"},{"key":"e_1_3_2_1_28_1","volume-title":"How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. CoRR abs\/1806.08295","author":"Colas C\u00e9dric","year":"2018","unstructured":"C\u00e9dric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. 2018. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. CoRR abs\/1806.08295 (2018). arXiv:1806.08295 http:\/\/arxiv.org\/abs\/1806.08295"},{"key":"e_1_3_2_1_29_1","volume-title":"A guide to deep learning in healthcare. Nature medicine 25, 1","author":"Esteva Andre","year":"2019","unstructured":"Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. 2019. A guide to deep learning in healthcare. Nature medicine 25, 1 (2019), 24--29."},{"key":"e_1_3_2_1_30_1","volume-title":"Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for Datasets. CoRR abs\/1803.09010 (2018). arXiv:1803.09010 http:\/\/arxiv.org\/abs\/1803.09010"},{"key":"e_1_3_2_1_31_1","volume-title":"ICSE '20: 42nd International Conference on Software Engineering","author":"Gerasimou Simos","year":"2020","unstructured":"Simos Gerasimou, Hasan Ferit Eniser, Alper Sen, and Alper Cakan. 2020. Importance-driven deep learning system testing. In ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 702--713."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Sindhu Ghanta Lior Khermosh Sriram Subramanian Vinay Sridhar Swaminathan Sundararaman Dulcardo Arteaga Qianmei Luo Drew Roselli Dhananjoy Das and Nisha Talagala. 2018. A systems perspective to reproducibility in production machine learning domain. (2018).","DOI":"10.1109\/ICMLA.2018.00105"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/103162.103163"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21918"},{"key":"e_1_3_2_1_35_1","volume-title":"Detecting Deep Neural Network Defects with Data Flow Analysis. In 51st Annual IEEE\/IFIP International Conference on Dependable Systems and Networks Workshops, DSN Workshops 2021","author":"Gu Jiazhen","year":"2021","unstructured":"Jiazhen Gu, Huanlin Xu, Haochuan Lu, Yangfan Zhou, and Xin Wang. 2021. Detecting Deep Neural Network Defects with Data Flow Analysis. In 51st Annual IEEE\/IFIP International Conference on Dependable Systems and Networks Workshops, DSN Workshops 2021, Taipei, Taiwan, June 21--24, 2021."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_2_1_37_1","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18)","author":"Gundersen Odd Erik","unstructured":"Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. [n.d.]. State of the Art: Reproducibility in Artificial Intelligence. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18)."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00080"},{"key":"e_1_3_2_1_39_1","volume-title":"Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. IEEE Computer Society, 770--778."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.359.6377.725"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Frank Hutter Lars Kotthoff and Joaquin Vanschoren (Eds.). 2019. Automated Machine Learning - Methods Systems Challenges. Springer. 10.1007\/978-3-030-05318-5","DOI":"10.1007\/978-3-030-05318-5"},{"key":"e_1_3_2_1_43_1","volume-title":"Asset Management in Machine Learning: A Survey. In 43rd IEEE\/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2021","author":"Idowu Samuel","year":"2021","unstructured":"Samuel Idowu, Daniel Str\u00fcber, and Thorsten Berger. 2021. Asset Management in Machine Learning: A Survey. In 43rd IEEE\/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2021, Madrid, Spain, May 25--28, 2021. IEEE, 51--60."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2019.00017"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2451116.2451118"},{"key":"e_1_3_2_1_46_1","volume-title":"Hinton","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 2015. Deep learning. Nat. (2015)."},{"key":"e_1_3_2_1_47_1","volume-title":"Proc. IEEE","author":"Lecun Y.","year":"1998","unstructured":"Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE (1998)."},{"key":"e_1_3_2_1_48_1","volume-title":"ICLR 2019 Workshop","author":"Lee Brian","year":"2019","unstructured":"Brian Lee, Andrew Jackson, Tom Madams, Seth Troisi, and Derek Jones. 2019. Minigo: A Case Study in Reproducing Reinforcement Learning Research. In Reproducibility in Machine Learning, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net."},{"key":"e_1_3_2_1_49_1","unstructured":"Wei Li. 2017. cifar-10-cnn: Play deep learning with CIFAR datasets. https:\/\/github.com\/BIGBALLON\/cifar-10-cnn."},{"key":"e_1_3_2_1_50_1","volume-title":"On the Replicability and Reproducibility of Deep Learning in Software Engineering. CoRR abs\/2006.14244","author":"Liu Chao","year":"2020","unstructured":"Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John C. Grundy, and Xiaohu Yang. 2020. On the Replicability and Reproducibility of Deep Learning in Software Engineering. CoRR abs\/2006.14244 (2020). arXiv:2006.14244 https:\/\/arxiv.org\/abs\/2006.14244"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132724"},{"key":"e_1_3_2_1_55_1","volume-title":"High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024--8035."},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416545"},{"key":"e_1_3_2_1_57_1","unstructured":"Joelle Pineau Philippe Vincent-Lamarre Koustuv Sinha Vincent Larivi\u00e8re Alina Beygelzimer Florence d'Alch\u00e9 Buc Emily Fox and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003.12206 [cs.LG]"},{"key":"e_1_3_2_1_58_1","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","author":"Raff Edward","year":"2019","unstructured":"Edward Raff. 2019. A Step Toward Quantifying Independently Reproducible Machine Learning Research. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.)."},{"key":"e_1_3_2_1_59_1","unstructured":"Jeanine Romano Jeffrey D Kromrey Jesse Coraggio and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research Vol. 13."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063384.2063405"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1200"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742833"},{"key":"e_1_3_2_1_63_1","volume-title":"Building a reproducible machine learning pipeline. arXiv preprint arXiv:1810.04570","author":"Sugimura Peter","year":"2018","unstructured":"Peter Sugimura and Florian Hartl. 2018. Building a reproducible machine learning pipeline. arXiv preprint arXiv:1810.04570 (2018)."},{"key":"e_1_3_2_1_64_1","volume-title":"Building a Reproducible Machine Learning Pipeline. CoRR abs\/1810.04570","author":"Sugimura Peter","year":"2018","unstructured":"Peter Sugimura and Florian Hartl. 2018. Building a Reproducible Machine Learning Pipeline. CoRR abs\/1810.04570 (2018). arXiv:1810.04570 http:\/\/arxiv.org\/abs\/1810.04570"},{"key":"e_1_3_2_1_65_1","unstructured":"Rachael Tatman Jake VanderPlas and Sohier Dane. 2018. A practical taxonomy of reproducibility for machine learning research. (2018)."},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2017.12.043"},{"key":"e_1_3_2_1_67_1","volume-title":"Open science is a research accelerator. Nature chemistry 3, 10","author":"Woelfle Michael","year":"2011","unstructured":"Michael Woelfle, Piero Olliaro, and Matthew H Todd. 2011. Open science is a research accelerator. Nature chemistry 3, 10 (2011), 745--748."},{"key":"e_1_3_2_1_68_1","unstructured":"Curtis Yanko. 2021 (accessed August 2021). Using a Software Bill of Materials (SBOM) is Going Mainstream. https:\/\/blog.sonatype.com\/software-bill-of-materials-going-mainstream"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.5244\/C.30.87"}],"event":{"name":"ICSE '22: 44th International Conference on Software Engineering","location":"Pittsburgh Pennsylvania","acronym":"ICSE '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 44th International Conference on Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510003.3510163","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3510003.3510163","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:10:25Z","timestamp":1750183825000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3510003.3510163"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,21]]},"references-count":69,"alternative-id":["10.1145\/3510003.3510163","10.1145\/3510003"],"URL":"https:\/\/doi.org\/10.1145\/3510003.3510163","relation":{},"subject":[],"published":{"date-parts":[[2022,5,21]]},"assertion":[{"value":"2022-07-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}