{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:42:14Z","timestamp":1791459734478,"version":"4.3.4"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100016135","name":"Universit\u00e4t Passau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016135","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Differential testing is a useful approach that uses different implementations of the same algorithms and compares the results for software testing. In recent years, this approach was successfully used for test campaigns of deep learning frameworks. There is little knowledge about the application of differential testing beyond deep learning. Within this article, we want to close this gap for classification algorithms. We conduct a case study using Scikit-learn, Weka, Spark MLlib, and Caret in which we identify the potential of differential testing by considering which algorithms are available in multiple frameworks, the feasibility by identifying pairs of algorithms that should exhibit the same behavior, and the effectiveness by executing tests for the identified pairs and analyzing the deviations. While we found a large potential for popular algorithms, the feasibility seems limited because, often, it is not possible to determine configurations that are the same in other frameworks. The execution of the feasible tests revealed that there is a large number of deviations for the scores and classes. Only a lenient approach based on statistical significance of classes does not lead to a huge amount of test failures. The potential of differential testing beyond deep learning seems limited for research into the quality of machine learning libraries. Practitioners may still use the approach if they have deep knowledge about implementations, especially if a coarse oracle that only considers significant differences of classes is sufficient.<\/jats:p>","DOI":"10.1007\/s10664-022-10273-9","type":"journal-article","created":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T15:25:48Z","timestamp":1674573948000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Differential testing for machine learning: an analysis for classification algorithms beyond deep learning"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9765-2803","authenticated-orcid":false,"given":"Steffen","family":"Herbold","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steffen","family":"Tunkel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,1,24]]},"reference":[{"key":"10273_CR1","unstructured":"Abadi M, Agarwal A , Barham P, Brevdo E , Chen Z , Citro C , Corrado GS, Davis A , Dean J , Devin M , Ghemawat S, Goodfellow I , Harp A, Irving G , Isard M , Jia Y , Jozefowicz R , Kaiser L , Kudlur M , Levenberg J , Man\u00e9 D , Monga R , Moore S , Murray D , Olah C , Schuster M, Shlens J, Steiner B , Sutskever I , Talwar K , Tucker P, Vanhoucke V, Vasudevan V , Vi\u00e9gas F , Vinyals O , Warden P , Wattenberg M , Wicke M, Yu Y, Zheng X (2015) TensorFlow : large-scale machine learning on heterogeneous systems https:\/\/www.tensorflow.org\/softwareavailablefromtensorflow.org"},{"key":"10273_CR2","unstructured":"Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operating systems design and implementation. OSDI\u201916, USENIX Association, Berkeley, pp 265\u2013283. http:\/\/dl.acm.org\/citation.cfm?id=3026877.3026899"},{"key":"10273_CR3","doi-asserted-by":"crossref","unstructured":"Asyrofi MH, Thung F, Lo D, Jiang L (2020) Crossasr: efficient differential testing of automatic speech recognition via text-to-speech","DOI":"10.1109\/ICSME46990.2020.00066"},{"key":"10273_CR4","doi-asserted-by":"publisher","unstructured":"Barash G, Farchi E, Jayaraman I, Raz O, Tzoref-Brill R, Zalmanovici M (2019) Bridging the gap between ml solutions and their business requirements using feature interactions. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. Association for Computing Machinery, New York, ESEC\/FSE 2019, pp 1048\u20131058 . https:\/\/doi.org\/10.1145\/3338906.3340442","DOI":"10.1145\/3338906.3340442"},{"key":"10273_CR5","unstructured":"Benavoli A, Corani G, Demsar J, Zaffalon M (2016) Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis. ArXiv:1606.04316"},{"key":"10273_CR6","doi-asserted-by":"publisher","first-page":"110542","DOI":"10.1016\/j.jss.2020.110542","volume":"164","author":"Braiek HB","year":"2020","unstructured":"Braiek HB, Khomh F (2020) On testing machine learning programs. J Syst Softw 164:110542. https:\/\/doi.org\/10.1016\/j.jss.2020.110542","journal-title":"J Syst Softw"},{"key":"10273_CR7","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1007\/BF00058655","volume":"24","author":"Breiman L","year":"1996","unstructured":"Breiman L (1996) Bagging predictors. Mach Learn 24:123\u2013140.","journal-title":"Mach Learn"},{"issue":"1","key":"10273_CR8","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/a:1010933404324","volume":"45","author":"Breiman L","year":"2001","unstructured":"Breiman L (2001) Random forests. Mach Learn 45(1):5\u201332. https:\/\/doi.org\/10.1023\/a:1010933404324","journal-title":"Mach Learn"},{"key":"10273_CR9","volume-title":"Classification and regression trees","author":"L Brieman","year":"1984","unstructured":"Brieman L, Friedman J, Olshen R, Stone C (1984) Classification and regression trees. Wadsworth, Belmont"},{"issue":"3","key":"10273_CR10","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1037\/0033-2909.114.3.494","volume":"114","author":"Cliff N","year":"1993","unstructured":"Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494\u2013509. https:\/\/doi.org\/10.1037\/0033-2909.114.3.494","journal-title":"Psychol Bull"},{"key":"10273_CR11","volume-title":"Quasi-experimentation: design & analysis issues for field settings, vol 351","author":"Cook TD","year":"1979","unstructured":"Cook TD, Campbell DT, Day A (1979) Quasi-experimentation: design & analysis issues for field settings, vol 351. Houghton Mifflin, Boston"},{"key":"10273_CR12","doi-asserted-by":"publisher","unstructured":"Davis MD, Weyuker EJ (1981) Pseudo-oracles for non-testable programs. In: Proceedings of the ACM \u201981 conference, ACM, New York, NY, USA, ACM \u201981, pp 254\u2013257, https:\/\/doi.org\/10.1145\/800175.809889","DOI":"10.1145\/800175.809889"},{"key":"10273_CR13","doi-asserted-by":"crossref","unstructured":"Ding J, Kang X, Hu XH (2017) Validating a deep learning framework by metamorphic testing. In: 2017 IEEE\/ACM 2nd international workshop on metamorphic testing (MET), pp 28\u201334, 10.1109\/MET.2017.2","DOI":"10.1109\/MET.2017.2"},{"key":"10273_CR14","unstructured":"Frank E, Hall MA, Witten IH (2016) The WEKA workbench online appendix for Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann"},{"issue":"1","key":"10273_CR15","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1006\/jcss.1997.1504","volume":"55","author":"Freund Y","year":"1997","unstructured":"Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119 \u2013 139. https:\/\/doi.org\/10.1006\/jcss.1997.1504","journal-title":"J Comput Syst Sci"},{"key":"10273_CR16","doi-asserted-by":"publisher","first-page":"111031","DOI":"10.1016\/j.jss.2021.111031","volume":"180","author":"Giray G","year":"2021","unstructured":"Giray G (2021) A software engineering perspective on engineering machine learning systems: state of the art and challenges. J Syst Softw 180:111031. https:\/\/doi.org\/10.1016\/j.jss.2021.111031","journal-title":"J Syst Softw"},{"issue":"3","key":"10273_CR17","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1109\/TSE.2013.59","volume":"40","author":"Groce A","year":"2014","unstructured":"Groce A, Kulesza T, Zhang C, Shamasunder S, Burnett M, Wong WK, Stumpf S, Das S, Shinsel A, Bice F, McIntosh K (2014) You are the only possible oracle: effective test selection for end users of interactive machine learning systems. IEEE Trans Softw Eng 40(3):307\u2013323. https:\/\/doi.org\/10.1109\/TSE.2013.59","journal-title":"IEEE Trans Softw Eng"},{"key":"10273_CR18","unstructured":"Gross P, Boulanger A, Arias M, Waltz D, Long PM, Lawson C, Anderson R, Koenig M, Mastrocinque M, Fairechio W, Johnson JA, Lee S, Doherty F, Kressner A (2006) Predicting electricity distribution feeder failures using machine learning susceptibility analysis. In: Proceedings of the 18th conference on innovative applications of artificial intelligence, vol 2. AAAI Press, IAAI\u201906, pp 1705\u20131711 http:\/\/dl.acm.org\/citation.cfm?id=1597122.1597127"},{"key":"10273_CR19","doi-asserted-by":"crossref","unstructured":"Guo Q, Xie X, Li Y, Zhang X, Liu Y, Li X, Shen C (2020) Audee: automated testing for deep learning frameworks. In: 2020 35th IEEE\/ACM international conference on automated software engineering (ASE), pp 486\u2013498","DOI":"10.1145\/3324884.3416571"},{"issue":"2","key":"10273_CR20","doi-asserted-by":"publisher","first-page":"933","DOI":"10.1109\/TNSE.2020.2997359","volume":"8","author":"Guo J","year":"2021","unstructured":"Guo J, Zhao Y, Song H, Jiang Y (2021) Coverage guided differential adversarial testing of deep learning systems. IEEE Trans Netw Sci Eng 8(2):933\u2013942. https:\/\/doi.org\/10.1109\/TNSE.2020.2997359","journal-title":"IEEE Trans Netw Sci Eng"},{"key":"10273_CR21","doi-asserted-by":"publisher","unstructured":"Herbold S, Haar T (2022) Smoke testing for machine learning: simple tests to discover severe bugs. Empir Softw Eng 27(2). https:\/\/doi.org\/10.1007\/s10664-021-10073-7","DOI":"10.1007\/s10664-021-10073-7"},{"key":"10273_CR22","doi-asserted-by":"publisher","unstructured":"ISO\/IEC\/IEEE (2017) Iso\/iec\/ieee international standard - systems and software engineering\u2013vocabulary. ISO\/IEC\/IEEE, 24765:2017(E) pp 1\u2013541. https:\/\/doi.org\/10.1109\/IEEESTD.2017.8016712","DOI":"10.1109\/IEEESTD.2017.8016712"},{"key":"10273_CR23","unstructured":"Karpathy A (2018) Cs231n: convolutional neural networks for visual recognition. https:\/\/cs231n.github.io\/neural-networks-3\/"},{"key":"10273_CR24","unstructured":"Kuhn M (2018) Caret: classification and regression training. https:\/\/CRAN.R-project.org\/package=caret,rpackageversion6.0-80"},{"issue":"1","key":"10273_CR25","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","volume":"18","author":"Mann HB","year":"1947","unstructured":"Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50 \u2013 60. https:\/\/doi.org\/10.1214\/aoms\/1177730491","journal-title":"Ann Math Stat"},{"key":"10273_CR26","doi-asserted-by":"crossref","unstructured":"Marijan D, Gotlieb A (2020) Software testing for machine learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34. pp 13576-13582","DOI":"10.1609\/aaai.v34i09.7084"},{"key":"10273_CR27","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez-Fern\u00e1ndez S, Bogner J, Franch X, Oriol M, Siebert J, Trendowicz A, Vollmer A M, Wagner S (2021) Software engineering for ai-based systems: a survey. ACM Trans Softw Eng Methodol","DOI":"10.1145\/3487043"},{"issue":"3","key":"10273_CR28","doi-asserted-by":"publisher","first-page":"e1279","DOI":"10.1002\/widm.1279","volume":"9","author":"McCullough BD","year":"2019","unstructured":"McCullough BD, Mokfi T, Almaeenejad M (2019) On the accuracy of linear regression routines in some data mining packages. WIREs Data Min Knowl Disc 9(3):e1279. https:\/\/doi.org\/10.1002\/widm.1279","journal-title":"WIREs Data Min Knowl Disc"},{"issue":"1","key":"10273_CR29","first-page":"1235","volume":"17","author":"X Meng","year":"2016","unstructured":"Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Freeman J, Tsai D, Amde M, Owen S et al (2016) Mllib: machine learning in apache spark. J Mach Learn Res 17(1):1235\u20131241","journal-title":"J Mach Learn Res"},{"key":"10273_CR30","unstructured":"Murphy C, Kaiser GE, Arias M (2007) An approach to software testing of machine learning applications. In: SEKE, vol 167"},{"key":"10273_CR31","unstructured":"Murphy C, Kaiser GE, Hu L, Wu L (2008) Properties of machine learning applications for use in metamorphic testing. In: SEKE, vol 8. pp 867-872"},{"key":"10273_CR32","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d'Alch\u00e9-Buc F, Fox E, Garnett R (eds) advances in neural information processing systems 32, Curran Associates, Inc, pp 8024\u20138035. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"10273_CR33","unstructured":"Patton MQ (2014) Qualitative research & evaluation methods: integrating theory and practice. Sage Publications"},{"key":"10273_CR34","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"issue":"11","key":"10273_CR35","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1145\/3361566","volume":"62","author":"Pei K","year":"2019","unstructured":"Pei K, Cao Y, Yang J, Jana S (2019) Deepxplore: automated whitebox testing of deep learning systems. Commun ACM 62(11):137\u2013145. https:\/\/doi.org\/10.1145\/3361566","journal-title":"Commun ACM"},{"key":"10273_CR36","doi-asserted-by":"publisher","unstructured":"Pham HV, Lutellier T, Qi W, Tan L (2019) Cradle: cross-backend validation to detect and localize bugs in deep learning libraries. In: 2019 IEEE\/ACM 41st international conference on software engineering (ICSE), pp 1027\u20131038. https:\/\/doi.org\/10.1109\/ICSE.2019.00107","DOI":"10.1109\/ICSE.2019.00107"},{"issue":"1","key":"10273_CR37","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1007\/BF00116251","volume":"1","author":"JR Quinlan","year":"1986","unstructured":"Quinlan JR (1986) Induction of decision trees. Mach Learn 1 (1):81\u2013106","journal-title":"Mach Learn"},{"key":"10273_CR38","volume-title":"C4.5: programs for machine learning","author":"JR Quinlan","year":"1993","unstructured":"Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc, San Francisco"},{"issue":"2","key":"10273_CR39","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","volume":"14","author":"P Runeson","year":"2009","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131\u2013164","journal-title":"Empir Softw Eng"},{"key":"10273_CR40","unstructured":"Theano Development Team (2016) Theano: a Python framework for fast computation of mathematical expressions. arXiv:1605.02688"},{"key":"10273_CR41","doi-asserted-by":"publisher","unstructured":"Tunkel S, Herbold S (2022) Replication Kit for: differential testing for machine learning: an analysis for classification algorithms beyond deep learning. https:\/\/doi.org\/10.5281\/zenodo.7341092","DOI":"10.5281\/zenodo.7341092"},{"key":"10273_CR42","doi-asserted-by":"publisher","unstructured":"Wang Z, Yan M, Chen J, Liu S, Zhang D (2020) Deep learning library testing via effective model generation. In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, Association for Computing Machinery, New York, NY, USA, ESEC\/FSE 2020, pp 788\u2013799. https:\/\/doi.org\/10.1145\/3368089.3409761","DOI":"10.1145\/3368089.3409761"},{"issue":"6","key":"10273_CR43","doi-asserted-by":"publisher","first-page":"80","DOI":"10.2307\/3001968","volume":"1","author":"Wilcoxon F","year":"1945","unstructured":"Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bulletin 1(6):80\u201383. http:\/\/www.jstor.org\/stable\/3001968","journal-title":"Biometrics Bulletin"},{"key":"10273_CR44","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in Software Engineering","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wesslen A (2012) Experimentation in Software Engineering. Springer Publishing Company, Incorporated"},{"issue":"4","key":"10273_CR45","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.jss.2010.11.920","volume":"84","author":"Xie X","year":"2011","unstructured":"Xie X, Ho JW, Murphy C, Kaiser G, Xu B, Chen TY (2011) Testing and validating machine learning classifiers by metamorphic testing. J Syst Softw 84(4):544\u2013558,. https:\/\/doi.org\/10.1016\/j.jss.2010.11.920","journal-title":"J Syst Softw"},{"key":"10273_CR46","unstructured":"Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX conference on hot topics in cloud computing (HotCloud)"},{"key":"10273_CR47","doi-asserted-by":"crossref","unstructured":"Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: survey , landscapes and horizons. IEEE Trans Softw Eng, pp 1\u20131","DOI":"10.1109\/TSE.2019.2962027"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10273-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10273-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10273-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,3]],"date-time":"2023-04-03T06:59:09Z","timestamp":1680505149000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10273-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,24]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["10273"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10273-9","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,24]]},"assertion":[{"value":"6 December 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 January 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"34"}}