{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T19:42:41Z","timestamp":1783194161473,"version":"3.54.6"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T00:00:00Z","timestamp":1642723200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T00:00:00Z","timestamp":1642723200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100018933","name":"Technische Universit\u00e4t Clausthal","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100018933","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic and simple smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that considerations how they can be adapted to this new context can help to find and prevent severe bugs, evenin mature machine learning libraries.<\/jats:p>","DOI":"10.1007\/s10664-021-10073-7","type":"journal-article","created":{"date-parts":[[2022,1,21]],"date-time":"2022-01-21T10:03:06Z","timestamp":1642759386000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Smoke testing for machine learning: simple tests to discover severe bugs"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9765-2803","authenticated-orcid":false,"given":"Steffen","family":"Herbold","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tobias","family":"Haar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,1,21]]},"reference":[{"key":"10073_CR1","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. https:\/\/www.tensorflow.org\/, Software available from tensorflow.org"},{"key":"10073_CR2","unstructured":"Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) Tensorflow: A system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI\u201916. USENIX Association, Berkeley, pp 265\u2013283. http:\/\/dl.acm.org\/citation.cfm?id=3026877.3026899"},{"key":"10073_CR3","unstructured":"Baltes S, Ralph P (2020) Sampling in software engineering research: A critical review and guidelines. arXiv:2002.07764"},{"issue":"10","key":"10073_CR4","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1145\/2544173.2509526","volume":"48","author":"T Bao","year":"2013","unstructured":"Bao T, Zhang X (2013) On-the-fly detection of instability problems in floating-point program execution. SIGPLAN Not 48(10):817\u2013832. https:\/\/doi.org\/10.1145\/2544173.2509526","journal-title":"SIGPLAN Not"},{"key":"10073_CR5","doi-asserted-by":"publisher","first-page":"110542","DOI":"10.1016\/j.jss.2020.110542","volume":"164","author":"HB Braiek","year":"2020","unstructured":"Braiek HB, Khomh F (2020) On testing machine learning programs. J Syst Softw 164:110542. https:\/\/doi.org\/10.1016\/j.jss.2020.110542, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121220300248","journal-title":"J Syst Softw"},{"key":"10073_CR6","unstructured":"Breck E, Zinkevich M, Polyzotis N, Whang S, Roy S (2019) Data validation for machine learning. In: Proceedings of SysML"},{"key":"10073_CR7","doi-asserted-by":"crossref","unstructured":"Chandrasekaran J, Feng H, Lei Y, Kuhn DR, Kacker R (2017) Applying combinatorial testing to data mining algorithms. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp 253\u2013261","DOI":"10.1109\/ICSTW.2017.46"},{"key":"10073_CR8","doi-asserted-by":"publisher","unstructured":"Collobert R, Weston J (2008) A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, ICML \u201908. ACM, New York, pp 160\u2013167. https:\/\/doi.org\/10.1145\/1390156.1390177","DOI":"10.1145\/1390156.1390177"},{"key":"10073_CR9","unstructured":"Dheeru D, KarraTaniskidou E (2017) UCI machine learning repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"10073_CR10","doi-asserted-by":"crossref","unstructured":"Ding J, Kang X, Hu XH (2017) Validating a deep learning framework by metamorphic testing. In: 2017 IEEE\/ACM 2nd International Workshop on Metamorphic Testing (MET), pp 28\u201334","DOI":"10.1109\/MET.2017.2"},{"key":"10073_CR11","unstructured":"Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: A deep convolutional activation feature for generic visual recognition. In: Xing EP, Jebara T (eds) Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research. http:\/\/proceedings.mlr.press\/v32\/donahue14.html, vol 32. PMLR, Bejing, pp 647\u2013655"},{"key":"10073_CR12","unstructured":"Frank E, Hall MA, Witten IH (2016) The weka workbench. online appendix for \u201ddata mining: Practical machine learning tools and techniques. Morgan Kaufmann"},{"key":"10073_CR13","doi-asserted-by":"publisher","unstructured":"Glance N, Hurst M, Nigam K, Siegler M, Stockton R, Tomokiyo T (2005) Deriving marketing intelligence from online discussion. In: Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, KDD \u201905. ACM, New York, pp 419\u2013428. https:\/\/doi.org\/10.1145\/1081870.1081919","DOI":"10.1145\/1081870.1081919"},{"issue":"3","key":"10073_CR14","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1109\/TSE.2013.59","volume":"40","author":"A Groce","year":"2014","unstructured":"Groce A, Kulesza T, Zhang C, Shamasunder S, Burnett M, Wong W-K, Stumpf S, Das S, Shinsel A, Bice F, McIntosh K (2014) You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Trans Softw Eng 40(3):307\u2013323. https:\/\/doi.org\/10.1109\/TSE.2013.59","journal-title":"IEEE Trans Softw Eng"},{"issue":"1","key":"10073_CR15","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1145\/1656274.1656278","volume":"11","author":"M Hall","year":"2009","unstructured":"Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10\u201318. https:\/\/doi.org\/10.1145\/1656274.1656278","journal-title":"ACM SIGKDD Explor Newslett"},{"issue":"7","key":"10073_CR16","doi-asserted-by":"publisher","first-page":"650","DOI":"10.1109\/TSE.2014.2327020","volume":"40","author":"C Henard","year":"2014","unstructured":"Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, LeTraon Y (2014) Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650\u2013670","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"10073_CR17","doi-asserted-by":"publisher","first-page":"847","DOI":"10.1016\/j.eswa.2006.07.007","volume":"33","author":"C-L Huang","year":"2007","unstructured":"Huang C-L, Chen M-C, Wang C-J (2007) Credit scoring with a data mining approach based on support vector machines. Expert Syst Appl 33 (4):847\u2013856. https:\/\/doi.org\/10.1016\/j.eswa.2006.07.007, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S095741740600217X","journal-title":"Expert Syst Appl"},{"key":"10073_CR18","doi-asserted-by":"publisher","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (May 2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84\u201390. https:\/\/doi.org\/10.1145\/3065386","DOI":"10.1145\/3065386"},{"issue":"6","key":"10073_CR19","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1109\/TSE.2004.24","volume":"30","author":"DR Kuhn","year":"2004","unstructured":"Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418\u2013421. https:\/\/doi.org\/10.1109\/TSE.2004.24","journal-title":"IEEE Trans Softw Eng"},{"key":"10073_CR20","unstructured":"Kuhn M (2018) caret: Classification and regression training. https:\/\/CRAN.R-project.org\/package=caret, R package version 6.0-80"},{"key":"10073_CR21","doi-asserted-by":"crossref","unstructured":"Ma L, Juefei-Xu F, Xue M, Li B, Li L, Liu Y, Zhao J (2019) Deepct: Tomographic combinatorial testing for deep learning systems. In: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, pp 614\u2013618","DOI":"10.1109\/SANER.2019.8668044"},{"key":"10073_CR22","unstructured":"Maclaurin D, Duvenaud D, Adams RP (2015) Gradient-based hyperparameter optimization through reversible learning. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML\u201915. JMLR.org, pp 2113\u20132122"},{"key":"10073_CR23","doi-asserted-by":"publisher","unstructured":"Murphy C, Kaiser G, Arias M (2007) Parameterizing random test data according to equivalence classes. In: Proceedings of the 2Nd International Workshop on Random Testing: Co-located with the 22Nd IEEE\/ACM International Conference on Automated Software Engineering (ASE 2007), RT \u201907. ACM, New York, pp 38\u201341. https:\/\/doi.org\/10.1145\/1292414.1292425","DOI":"10.1145\/1292414.1292425"},{"key":"10073_CR24","unstructured":"Murphy C, Kaiser GE, Hu L, Wu L (2008) Properties of machine learning applications for use in metamorphic testing. In: SEKE, vol 8, pp 867\u2013872"},{"key":"10073_CR25","doi-asserted-by":"crossref","unstructured":"Nakajima S, Bui HN (2016) Dataset coverage for testing machine learning computer programs. In: 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), pp 297\u2013304","DOI":"10.1109\/APSEC.2016.049"},{"key":"10073_CR26","doi-asserted-by":"crossref","unstructured":"Nejadgholi M, Yang J (2019) A study of oracle approximations in testing deep learning libraries. In: 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp 785\u2013796","DOI":"10.1109\/ASE.2019.00078"},{"key":"10073_CR27","doi-asserted-by":"crossref","unstructured":"Paek T, Gamon M, Counts S, Chickering DM, Dhesi A (2010) Predicting the importance of newsfeed posts and social network friends. In: Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI\u201910. AAAI Press, pp 1419\u20131424. http:\/\/dl.acm.org\/citation.cfm?id=2898607.2898833","DOI":"10.1609\/aaai.v24i1.7518"},{"key":"10073_CR28","unstructured":"Patton MQ (2014) Qualitative research & evaluation methods: Integrating theory and practice. Sage publications"},{"key":"10073_CR29","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine learning in Python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"key":"10073_CR30","doi-asserted-by":"publisher","unstructured":"Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, SOSP \u201917. ACM, New York, pp 1\u201318. https:\/\/doi.org\/10.1145\/3132747.3132785","DOI":"10.1145\/3132747.3132785"},{"key":"10073_CR31","doi-asserted-by":"crossref","unstructured":"Pham HV, Lutellier T, Qi W, Tan L (2019) Cradle: Cross-backend validation to detect and localize bugs in deep learning libraries. In: 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE), pp 1027\u20131038","DOI":"10.1109\/ICSE.2019.00107"},{"key":"10073_CR32","unstructured":"Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. In: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2, NIPS\u201912. Curran Associates Inc., Red Hook, pp 2951\u20132959"},{"key":"10073_CR33","doi-asserted-by":"publisher","unstructured":"Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE \u201918. ACM, New York, pp 303\u2013314. https:\/\/doi.org\/10.1145\/3180155.3180220","DOI":"10.1145\/3180155.3180220"},{"key":"10073_CR34","doi-asserted-by":"crossref","unstructured":"Udeshi S, Arora P, Chattopadhyay S (2018) Automated directed fairness testing. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, pp 98\u2013108","DOI":"10.1145\/3238147.3238165"},{"key":"10073_CR35","doi-asserted-by":"publisher","unstructured":"Wang Z, Yan M, Chen J, Liu S, Zhang D (2020) Deep learning library testing via effective model generation. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020. Association for Computing Machinery, New York, pp 788\u2013799. https:\/\/doi.org\/10.1145\/3368089.3409761","DOI":"10.1145\/3368089.3409761"},{"key":"10073_CR36","doi-asserted-by":"crossref","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wesslen A (2012) Experimentation in Software Engineering. Springer Publishing Company, Incorporated","DOI":"10.1007\/978-3-642-29044-2"},{"key":"10073_CR37","doi-asserted-by":"publisher","unstructured":"Xie X, Chen H, Li Y, Ma L, Liu Y, Zhao J (2019) Coverage-guided fuzzing for feedforward neural networks. In: Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering, ASE \u201919. IEEE Press, pp 1162\u20131165. https:\/\/doi.org\/10.1109\/ASE.2019.00127","DOI":"10.1109\/ASE.2019.00127"},{"issue":"4","key":"10073_CR38","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.jss.2010.11.920","volume":"84","author":"X Xie","year":"2011","unstructured":"Xie X, Ho JWK, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84(4):544\u2013558. https:\/\/doi.org\/10.1016\/j.jss.2010.11.920, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121210003213, The Ninth International Conference on Quality Software","journal-title":"J Syst Softw"},{"key":"10073_CR39","unstructured":"Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud)"},{"key":"10073_CR40","doi-asserted-by":"crossref","unstructured":"Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: Survey, landscapes and horizons. IEEE Trans Softw Eng:1\u20131","DOI":"10.1109\/TSE.2019.2962027"},{"key":"10073_CR41","unstructured":"Zhang JM, Barr ET, Guedj B, Harman M, Shawe-Taylor J (2019) Perturbed model validation: A new framework to validate model relevance. CoRR arXiv:1905.10201"},{"issue":"4","key":"10073_CR42","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1007\/s11767-012-0743-9","volume":"28","author":"J Zhang","year":"2011","unstructured":"Zhang J, Hu X, Zhang B (2011) An evaluation approach for the program of association rules algorithm based on metamorphic relations. J Electron (China) 28(4):623\u2013631. https:\/\/doi.org\/10.1007\/s11767-012-0743-9","journal-title":"J Electron (China)"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-021-10073-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-021-10073-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-021-10073-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:38:24Z","timestamp":1674520704000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-021-10073-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,21]]},"references-count":42,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,3]]}},"alternative-id":["10073"],"URL":"https:\/\/doi.org\/10.1007\/s10664-021-10073-7","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,21]]},"assertion":[{"value":"28 October 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 January 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"45"}}