{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T03:21:22Z","timestamp":1784085682067,"version":"3.55.0"},"reference-count":43,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Austrian Ministry for Transport, Innovation, and Technology (BMVIT)","award":["888127"],"award-info":[{"award-number":["888127"]}]},{"name":"Federal Ministry for Digital and Economic Affairs (BMDW)","award":["888127"],"award-info":[{"award-number":["888127"]}]},{"name":"Province of Upper Austria in the frame of the COMET-Competence Centers for Excellent Technologies Program","award":["888127"],"award-info":[{"award-number":["888127"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>In the last decade, industry\u2019s demand for deep learning (DL) has increased due to its high performance in complex scenarios. Due to the DL method\u2019s complexity, experts and non-experts rely on blackbox software packages such as Tensorflow and Pytorch. The frameworks are constantly improving, and new versions are released frequently. As a natural process in software development, the released versions contain improvements\/changes in the methods and their implementation. Moreover, versions may be bug-polluted, leading to the model performance decreasing or stopping the model from working. The aforementioned changes in implementation can lead to variance in obtained results. This work investigates the effect of implementation changes in different major releases of these frameworks on the model performance. We perform our study using a variety of standard datasets. Our study shows that users should consider that changing the framework version can affect the model performance. Moreover, they should consider the possibility of a bug-polluted version before starting to debug source code that had an excellent performance before a version change. This also shows the importance of using virtual environments, such as Docker, when delivering a software product to clients.<\/jats:p>","DOI":"10.3390\/make4040045","type":"journal-article","created":{"date-parts":[[2022,10,9]],"date-time":"2022-10-09T01:43:11Z","timestamp":1665279791000},"page":"888-911","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models?"],"prefix":"10.3390","volume":"4","author":[{"given":"Mostafa","family":"Shahriari","sequence":"first","affiliation":[{"name":"Software Competence Center Hagenberg GmbH (SCCH), 4232 Hagenberg, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9903-6107","authenticated-orcid":false,"given":"Rudolf","family":"Ramler","sequence":"additional","affiliation":[{"name":"Software Competence Center Hagenberg GmbH (SCCH), 4232 Hagenberg, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5303-6638","authenticated-orcid":false,"given":"Lukas","family":"Fischer","sequence":"additional","affiliation":[{"name":"Software Competence Center Hagenberg GmbH (SCCH), 4232 Hagenberg, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"56","DOI":"10.3390\/make3010004","article-title":"AI System Engineering\u2014Key Challenges and Lessons Learned","volume":"3","author":"Fischer","year":"2021","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_2","unstructured":"Lu, L., Zheng, Y., Carneiro, G., and Yang, L. (2017). Deep Learning for Computer Vision: Expert Techniques to Train Advanced Neural Networks Using TensorFlow and Keras, Springer."},{"key":"ref_3","unstructured":"Yu, D., and Deng, L. (2017). Automatic Speech Recognition: A Deep Learning Approach, Springer."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bhanu, B., and Kumar, A. (2017). Deep Learning for Biometrics, Springer.","DOI":"10.1007\/978-3-319-61657-5"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"D83","DOI":"10.1190\/geo2021-0240.1","article-title":"A deep learning approach to design a borehole instrument for geosteering","volume":"87","author":"Shahriari","year":"2022","journal-title":"Geophysics"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1629","DOI":"10.1002\/nme.6593","article-title":"Error control and loss functions for the deep learning inversion of borehole resistivity measurements","volume":"122","author":"Shahriari","year":"2021","journal-title":"Int. J. Numer. Methods Eng."},{"key":"ref_7","first-page":"860","article-title":"Deep learning: An introduction for applied mathematicians","volume":"61","author":"Higham","year":"2018","journal-title":"Comput. Res. Repos."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Eiben, A., and Smith, J. (2015). Introduction to Evolutionary Computing, Springer.","DOI":"10.1007\/978-3-662-44874-8"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Debnath, P., and Mohiuddine, S. (2021). Soft Computing Techniques in Engineering, Health, Mathematical and Social Sciences, CRC Press.","DOI":"10.1201\/9781003161707"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Debnath, P., Castillo, O., and Kumam, P. (2023). Soft Computing: Recent Advances and Applications in Engineering and Mathematical Sciences, CRC Press.","DOI":"10.1201\/9781003312017"},{"key":"ref_11","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2022, August 01). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_12","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Somepalli, G., Fowl, L., Bansal, A., Yeh-Chiang, P., Dar, Y., Baraniuk, R., Goldblum, M., and Goldstein, T. (2022, January 18\u201324). Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA. Available online: http:\/\/xxx.lanl.gov\/abs\/2203.08124.","DOI":"10.1109\/CVPR52688.2022.01333"},{"key":"ref_14","unstructured":"Nagarajan, P., Warnell, G., and Stone, P. (2022, August 01). The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning. Available online: https:\/\/openreview.net\/forum?id=S1e-OsZ4e7."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"211860","DOI":"10.1109\/ACCESS.2020.3039833","article-title":"Challenges for the Repeatability of Deep Learning Models","volume":"8","author":"Alahmari","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Pham, H.V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., and Nagappan, N. (2020, January 21\u201325). Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance. Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering, Virtual. ASE \u201920.","DOI":"10.1145\/3324884.3416545"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"100073","DOI":"10.1016\/j.patter.2020.100073","article-title":"dtoolAI: Reproducibility for Deep Learning","volume":"1","author":"Hartley","year":"2020","journal-title":"Patterns"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1162\/tacl_a_00018","article-title":"Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results","volume":"6","author":"Crane","year":"2018","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Beam, A., Manrai, A., and Ghassemi, M. (2020). Challenges to the Reproducibility of Machine Learning Models in Health Care. JAMA, 323.","DOI":"10.1001\/jama.2019.20866"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.future.2021.06.014","article-title":"Do machine learning platforms provide out-of-the-box reproducibility?","volume":"126","author":"Gundersen","year":"2022","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_21","unstructured":"Gundersen, O.E., Coakley, K., and Kirkpatrick, C. (2022). Sources of Irreproducibility in Machine Learning: A Review. arXiv."},{"key":"ref_22","unstructured":"(2022, August 01). NVIDIA; Vingelmann, P.; Fitzek, F.H.P. CUDA, Release: 10.2.89. Available online: https:\/\/developer.nvidia.com\/cuda-toolkit."},{"key":"ref_23","unstructured":"Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E. (2014). cuDNN: Efficient Primitives for Deep Learning. arXiv, Available online: http:\/\/xxx.lanl.gov\/abs\/1410.0759."},{"key":"ref_24","unstructured":"Struski, L., Morkisz, P., Spurek, P., Bernabeu, S.R., and Trzcinski, T. (2021). Efficient GPU implementation of randomized SVD and its applications. arXiv, Available online: http:\/\/xxx.lanl.gov\/abs\/2110.03423."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"20167","DOI":"10.1073\/pnas.0709640104","article-title":"Randomized algorithms for the low-rank approximation of matrices","volume":"104","author":"Liberty","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Rivera-Landos, E., Khomh, F., and Nikanjam, A. (2021, January 6\u201310). The Challenge of Reproducible ML: An Empirical Study on The Impact of Bugs. Proceedings of the IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), Hainan, China.","DOI":"10.1109\/QRS54544.2021.00116"},{"key":"ref_27","unstructured":"Tambon, F., Nikanjam, A., An, L., Khomh, F., and Antoniol, G. (2021). Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Leotta, M., Olianas, D., Ricca, F., and Noceti, N. (2019, January 8\u201312). How Do Implementation Bugs Affect the Results of Machine Learning Algorithms?. Proceedings of the 34th ACM\/SIGAPP Symposium on Applied Computing, Limassol, Cyprus. SAC \u201919.","DOI":"10.1145\/3297280.3297411"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Chen, Y., Cheung, S.C., Xiong, Y., and Zhang, L. (2018, January 16\u201321). An Empirical Study on TensorFlow Program Bugs. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Amsterdam, The Netherlands. ISSTA 2018.","DOI":"10.1145\/3213846.3213866"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Dwarakanath, A., Ahuja, M., Sikand, S., Rao, R.M., Bose, R.P.J.C., Dubash, N., and Podder, S. (2018, January 16\u201321). Identifying Implementation Bugs in Machine Learning Based Image Classifiers Using Metamorphic Testing. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Amsterdam, The Netherlands. ISSTA 2018.","DOI":"10.1145\/3213846.3213858"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., and Tonella, P. (2020, January 5\u201311). Taxonomy of Real Faults in Deep Learning Systems. Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering, Seoul, Korea. ICSE \u201920.","DOI":"10.1145\/3377811.3380395"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Islam, M.J., Nguyen, G., Pan, R., and Rajan, H. (2019, January 26\u201330). A Comprehensive Study on Deep Learning Bug Characteristics. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn, Estonia. ESEC\/FSE 2019.","DOI":"10.1145\/3338906.3338955"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic Minority Over-sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res. (JAIR)"},{"key":"ref_34","first-page":"56","article-title":"On reproducible AI: Towards reproducible research, open science, and digital scholarship in AI publications","volume":"39","author":"Gundersen","year":"2018","journal-title":"AI Mag."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Gundersen, O.E., and Kjensmo, S. (2018, January 2\u20137). State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, Orleans, France.","DOI":"10.1609\/aaai.v32i1.11503"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"E14","DOI":"10.1038\/s41586-020-2766-y","article-title":"Transparency and reproducibility in artificial intelligence","volume":"586","author":"Adam","year":"2020","journal-title":"Nature"},{"key":"ref_37","first-page":"316","article-title":"Randomness in neural network training: Characterizing the impact of tooling","volume":"4","author":"Zhuang","year":"2022","journal-title":"Proc. Mach. Learn. Syst."},{"key":"ref_38","first-page":"30211","article-title":"Are my deep learning systems fair? An empirical study of fixed-seed training","volume":"34","author":"Qian","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Morovati, M.M., Nikanjam, A., Khomh, F., and Ming, Z. (2022). Bugs in Machine Learning-based Systems: A Faultload Benchmark. arXiv.","DOI":"10.1007\/s10664-023-10291-1"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"110935","DOI":"10.1016\/j.jss.2021.110935","article-title":"The symptoms, causes, and repairs of bugs inside a deep learning library","volume":"177","author":"Jia","year":"2021","journal-title":"J. Syst. Softw."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019, January 25\u201331). Software engineering for machine learning: A case study. Proceedings of the IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada.","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"111031","DOI":"10.1016\/j.jss.2021.111031","article-title":"A software engineering perspective on engineering machine learning systems: State of the art and challenges","volume":"180","author":"Giray","year":"2021","journal-title":"J. Syst. Softw."},{"key":"ref_43","unstructured":"(2022, August 01). Keras Tuner. Available online: https:\/\/github.com\/keras-team\/keras-tuner."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/45\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:47:02Z","timestamp":1760143622000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/4\/4\/45"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,5]]},"references-count":43,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["make4040045"],"URL":"https:\/\/doi.org\/10.3390\/make4040045","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,5]]}}}