{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T20:03:30Z","timestamp":1771272210108,"version":"3.50.1"},"reference-count":53,"publisher":"Walter de Gruyter GmbH","issue":"1","license":[{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,1,23]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>A rapid rise in machine learning-based applications has made it one of the most popular areas in the field of artificial intelligence (AI). The most commonly used libraries to implement the algorithms used in these applications are Scikit learn and Weka. It is challenging to test these machines learning based applications due to the Oracle Problem. The problem is when the expected outcome is not known and hence the testing of such applications cannot be performed via traditional testing techniques. One of the solution to the Oracle problem is the use of Metamorphic testing to test the machine learning applications. The code of machine learning algorithms is often ignored, when testing of ML-based applications is done. However, the usage of the machine learning algorithms within the libraries requires formal testing to improve reliability. This work evaluates the Metamorphic relations for machine learning algorithms by finding their kill rate while testing 5 machine learning (ANN, ID3, KNN, Naive Bayes, SVM) classifiers from the Scikit Learn library. This work also calculates the statement coverage, while testing the metamorphic relations. The relationship between the effectiveness of fault detection and code coverage is identified as\u00a0well.<\/jats:p>","DOI":"10.1515\/jisys-2024-0363","type":"journal-article","created":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T19:09:53Z","timestamp":1771268993000},"source":"Crossref","is-referenced-by-count":0,"title":["Performance of test cases for machine learning classifier: coverage perspective"],"prefix":"10.1515","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8689-904X","authenticated-orcid":false,"given":"Sadia","family":"Ashraf","sequence":"first","affiliation":[{"name":"66683 International Islamic University Islamabad , Islamabad , Pakistan"}]},{"given":"Salma","family":"Imtiaz","sequence":"additional","affiliation":[{"name":"66683 International Islamic University Islamabad , Islamabad , Pakistan"}]},{"given":"Asmat ullah","family":"Khan","sequence":"additional","affiliation":[{"name":"66683 International Islamic University Islamabad , Islamabad , Pakistan"},{"name":"Department of Computer Science , Prince Sultaan University , Riyadh , Saudia Arabia"}]},{"given":"Rastislav","family":"Kulhanek","sequence":"additional","affiliation":[{"name":"Faculty of Management , Comenius University Bratislava , Bratislava , Slovakia"}]}],"member":"374","published-online":{"date-parts":[[2026,2,16]]},"reference":[{"key":"2026021619094514479_j_jisys-2024-0363_ref_051","doi-asserted-by":"crossref","unstructured":"Z. Q. Zhou and L. Sun, \u201cMetamorphic testing of driverless cars,\u201d Commun. ACM, vol.\u00a062, no. 3, pp.\u00a061\u201367, 2019, https:\/\/doi.org\/10.1145\/3241979.","DOI":"10.1145\/3241979"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_035","doi-asserted-by":"crossref","unstructured":"J.\u00a0G. Richens, C. M. Lee, and S. Johri, \u201cImproving the accuracy of medical diagnosis with causal machine learning,\u201d Nat. Commun., vol.\u00a011, no. 1, p.\u00a03923, 2020, https:\/\/doi.org\/10.1038\/s41467-020-17419-7.","DOI":"10.1038\/s41467-020-17419-7"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_027","doi-asserted-by":"crossref","unstructured":"R. P. Masini, M. C. Medeiros, and E. F. Mendes, \u201cMachine learning advances for time series forecasting,\u201d J.\u00a0Econ. Surv., vol.\u00a037, no. 1, pp.\u00a076\u2013111, 2023, https:\/\/doi.org\/10.1111\/joes.12429.","DOI":"10.1111\/joes.12429"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_011","doi-asserted-by":"crossref","unstructured":"M. Lister, Fundamentals of Operating Systems, 3rd ed. New York, Springer-Verlag, 1984.","DOI":"10.1007\/978-1-349-17540-6"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_012","doi-asserted-by":"crossref","unstructured":"D. Gibert, C. Mateu, and J. Planes, \u201cThe rise of machine learning for detection and classification of malware: research developments, trends and challenges,\u201d J.\u00a0Netw. Comput. Appl., vol. 153, no. 1, p. 102526, 2020. https:\/\/doi.org\/10.1016\/j.jnca.2019.102526.","DOI":"10.1016\/j.jnca.2019.102526"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_048","doi-asserted-by":"crossref","unstructured":"Y. Yang, Z. Li, H. Wang, C. Xu, and X. Ma, \u201cTowards effective metamorphic testing by algorithm stability for linear classification programs,\u201d J.\u00a0Syst. Software, vol. 180, no. 4, p. 111012, 2021. https:\/\/doi.org\/10.1016\/j.jss.2021.111012.","DOI":"10.1016\/j.jss.2021.111012"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_001","unstructured":"S. Al-Azani and J. Hassine, \u201cValidation of machine learning classifiers using metamorphic testing and feature selection techniques,\u201d Paper Presented at the Multi-disciplinary Trends in Artificial Intelligence: 11Th International Workshop, MIWAI 2017, Gadong, Brunei, November 20\u201322, 2017, Proceedings 11, 2017."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_002","doi-asserted-by":"crossref","unstructured":"P. Ammann and J. Offutt, Introduction to Software Testing, Cambridge, UK, Cambridge University Press, 2016.","DOI":"10.1017\/9781316771273"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_018","doi-asserted-by":"crossref","unstructured":"H. Jin, Y. Wang, N.-W. Chen, Z.-J. Gou, and S. Wang, \u201cArtificial neural network for automatic test oracles generation,\u201d Paper presented at the 2008 International Conference on Computer Science and Software Engineering, 2008.","DOI":"10.1109\/CSSE.2008.774"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_036","doi-asserted-by":"crossref","unstructured":"P. Saha and U. Kanewala, \u201cFault detection effectiveness of metamorphic relations developed for testing supervised classifiers,\u201d Paper presented at the 2019 IEEE International conference on artificial intelligence testing (AITest), 2019.","DOI":"10.1109\/AITest.2019.00019"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_004","doi-asserted-by":"crossref","unstructured":"E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, \u201cThe oracle problem in software testing: a survey,\u201d IEEE Trans. Software Eng., vol.\u00a041, no. 5, pp.\u00a0507\u2013525, 2014, https:\/\/doi.org\/10.1109\/tse.2014.2372785.","DOI":"10.1109\/TSE.2014.2372785"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_016","doi-asserted-by":"crossref","unstructured":"T. Jameel, L. Mengxiang, and L. Chao, \u201cAutomatic test oracle for image processing applications using support vector machines,\u201d Paper presented at the 2015\u00a06th IEEE International Conference on Software Engineering and Service Science (ICSESS), 2015.","DOI":"10.1109\/ICSESS.2015.7339246"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_006","unstructured":"T. Y. Chen, S. C. Cheung, and S. M. Yiu, \u201cMetamorphic testing: a new approach for generating next test cases,\u201d arXiv preprint arXiv:2002.12543, 2020."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_007","doi-asserted-by":"crossref","unstructured":"T. Y. Chen et al.., \u201cMetamorphic testing: a review of challenges and opportunities,\u201d ACM Comput. Surv., vol.\u00a051, no. 1, pp.\u00a01\u201327, 2018.","DOI":"10.1145\/3143561"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_010","doi-asserted-by":"crossref","unstructured":"V. Garousi and M. V. M\u00e4ntyl\u00e4, \u201cA systematic literature review of literature reviews in software testing,\u201d Inf. Software Technol., vol. 80, no. 12, pp. 195\u2013216, 2016. https:\/\/doi.org\/10.1016\/j.infsof.2016.09.002.","DOI":"10.1016\/j.infsof.2016.09.002"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_023","doi-asserted-by":"crossref","unstructured":"P. Ma, S. Wang, and J. Liu, \u201cMetamorphic testing and certified mitigation of fairness violations in NLP models,\u201d Paper presented at the IJCAI, 2020.","DOI":"10.24963\/ijcai.2020\/64"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_045","doi-asserted-by":"crossref","unstructured":"X. Xie, J. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, \u201cApplication of metamorphic testing to supervised classifiers,\u201d Paper presented at the 2009 Ninth International Conference on Quality Software, 2009.","DOI":"10.1109\/QSIC.2009.26"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_046","unstructured":"X. Xie, J. Ho, C. Murphy, G. E. Kaiser, B. Xu, and T. Y. Chen, \u201cImproving the quality of computational science software by using metamorphic relations to test machine learning applications,\u201d 2009."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_047","doi-asserted-by":"crossref","unstructured":"X. Xie, J.\u00a0W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, \u201cTesting and validating machine learning classifiers by metamorphic testing,\u201d J.\u00a0Syst. Software, vol.\u00a084, no. 4, pp.\u00a0544\u2013558, 2011, https:\/\/doi.org\/10.1016\/j.jss.2010.11.920.","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_003","doi-asserted-by":"crossref","unstructured":"A. K. Ashraf, M. D\u2019Souza, and R. Jetley, \u201cCoverage criteria based testing of industrial robots,\u201d Paper presented at the 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), 2020.","DOI":"10.1109\/CASE48305.2020.9217031"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_013","doi-asserted-by":"crossref","unstructured":"A. Gupta and P. Jalote, \u201cAn approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing,\u201d Int. J.\u00a0Software Tool. Technol. Tran., vol. 10, no. 2, pp. 145\u2013160, 2008. https:\/\/doi.org\/10.1007\/s10009-007-0059-5.","DOI":"10.1007\/s10009-007-0059-5"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_031","doi-asserted-by":"crossref","unstructured":"S. Nakajima and H. N. Bui, \u201cDataset coverage for testing machine learning computer programs,\u201d Paper presented at the 2016\u00a023rd Asia-Pacific Software Engineering Conference (APSEC), 2016.","DOI":"10.1109\/APSEC.2016.049"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_033","doi-asserted-by":"crossref","unstructured":"R. Qian, Q. Zhang, C. Fang, and L. Guo, \u201cInvestigating coverage guided fuzzing with mutation testing,\u201d Paper presented at the Proceedings of the 13th Asia-Pacific Symposium on Internetware, 2022.","DOI":"10.1145\/3545258.3545285"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_026","doi-asserted-by":"crossref","unstructured":"A. Martin-Lopez, A. Arcuri, S. Segura, and A. Ruiz-Cort\u00e9s, \u201cBlack-box and white-box test case generation for RESTful APIs: enemies or allies?\u201d Paper presented at the 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), 2021.","DOI":"10.1109\/ISSRE52982.2021.00034"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_032","doi-asserted-by":"crossref","unstructured":"M. Naz, Z. Anwaar, and W. H. Butt, \u201cAutomated white box test case generation for statement coverage using U-NSGA-III,\u201d Paper presented at the 2023\u00a017th International Conference on Open Source Systems and Technologies (ICOSST), 2023.","DOI":"10.1109\/ICOSST60641.2023.10414201"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_029","unstructured":"S. Nakajima, \u201cGeneralized oracle for testing machine learning computer programs,\u201d Paper presented at the Software Engineering and Formal Methods: SEFM 2017 Collocated Workshops: DataMod, FAACS, MSE, CoSim-CPS, and FOCLASA, Trento, Italy, September 4\u20135, 2017, Revised Selected Papers 15, 2018."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_030","doi-asserted-by":"crossref","unstructured":"S. Nakajima, \u201cDataset diversity for metamorphic testing of machine learning software,\u201d Paper presented at the Structured Object-Oriented Formal Language and Method: 8th International Workshop, SOFL+ MSVL 2018, Gold Coast, QLD, Australia, November 16, 2018, Revised Selected Papers 8, 2019.","DOI":"10.1007\/978-3-030-13651-2_2"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_021","doi-asserted-by":"crossref","unstructured":"C. Liem, Y. X. Gu, and H. Johnson, \u201cA compiler-based infrastructure for software-protection,\u201d Paper presented at the Proceedings of the third ACM SIGPLAN workshop on Programming languages and analysis for security, 2008.","DOI":"10.1145\/1375696.1375702"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_038","doi-asserted-by":"crossref","unstructured":"S. Schrittwieser, S. Katzenbeisser, J. Kinder, G. Merzdovnik, and E. Weippl, \u201cProtecting software through obfuscation: can it keep pace with progress in code analysis?\u201d ACM Comput. Surv., vol.\u00a049, no. 1, pp.\u00a01\u201337, 2016, https:\/\/doi.org\/10.1145\/2886012.","DOI":"10.1145\/2886012"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_050","doi-asserted-by":"crossref","unstructured":"Q. Zhang, J. Wang, M. A. Gulzar, R. Padhye, and M. Kim, \u201cBigfuzz: efficient fuzz testing for data analytics using framework abstraction,\u201d Paper presented at the Proceedings of the 35th IEEE\/ACM international conference on automated software engineering, 2020.","DOI":"10.1145\/3324884.3416641"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_041","doi-asserted-by":"crossref","unstructured":"Z. Sun, J.\u00a0M. Zhang, M. Harman, M. Papadakis, and L. Zhang, \u201cAutomatic testing and improvement of machine translation,\u201d Paper presented at the Proceedings of the ACM\/IEEE 42nd international conference on software engineering, 2020.","DOI":"10.1145\/3377811.3380420"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_015","doi-asserted-by":"crossref","unstructured":"S. Jalil, S. Rafi, T. D. LaToza, K. Moran, and W. Lam, \u201cChatgpt and software testing education: promises & perils,\u201d Paper presented at the 2023 IEEE international conference on software testing, verification and validation workshops (ICSTW), 2023.","DOI":"10.1109\/ICSTW58534.2023.00078"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_014","doi-asserted-by":"crossref","unstructured":"K. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep residual learning for image recognition,\u201d Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.","DOI":"10.1109\/CVPR.2016.90"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_009","doi-asserted-by":"crossref","unstructured":"Y. Deng et al.., \u201cA declarative metamorphic testing framework for autonomous driving,\u201d IEEE Trans. Software Eng., vol.\u00a049, no. 4, pp.\u00a01964\u20131982, 2022, https:\/\/doi.org\/10.1109\/tse.2022.3206427.","DOI":"10.1109\/TSE.2022.3206427"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_049","doi-asserted-by":"crossref","unstructured":"M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid, \u201cDeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems,\u201d Paper presented at the Proceedings of the 33rd ACM\/IEEE international conference on automated software engineering, 2018.","DOI":"10.1145\/3238147.3238187"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_040","doi-asserted-by":"crossref","unstructured":"M. P. Shahri, M. Srinivasan, G. Reynolds, D. Bimczok, I. Kahanda, and U. Kanewala, \u201cMetamorphic testing for quality assurance of protein function prediction tools,\u201d in 2019 IEEE International Conference On Artificial Intelligence Testing (AITest), pp. 140\u2013148. IEEE, 2019.","DOI":"10.1109\/AITest.2019.00017"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_025","doi-asserted-by":"crossref","unstructured":"Y. Ma, D. Towey, T. Y. Chen, and Z. Q. Zhou, \u201cMetamorphic testing of fake news detection software,\u201d Paper presented at the 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), 2021.","DOI":"10.1109\/COMPSAC51774.2021.00224"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_017","doi-asserted-by":"crossref","unstructured":"M. Jiang, T. Y. Chen, and S. Wang, \u201cOn the effectiveness of testing sentiment analysis systems with metamorphic testing,\u201d Inf. Software Technol., vol. 150, no. 10, p. 106966, 2022. https:\/\/doi.org\/10.1016\/j.infsof.2022.106966.","DOI":"10.1016\/j.infsof.2022.106966"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_024","doi-asserted-by":"crossref","unstructured":"Y. Ma, Y. Pan, and Y. Fan, \u201cMetamorphic testing of classification program for the COVID-19 intelligent diagnosis,\u201d Paper presented at the 2022\u00a09th International Conference on Dependable Systems and Their Applications (DSA), 2022.","DOI":"10.1109\/DSA56465.2022.00031"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_043","doi-asserted-by":"crossref","unstructured":"F. ur Rehman and C. Izurieta, \u201cStatistical metamorphic testing of neural network based intrusion detection systems,\u201d Paper presented at the 2021 IEEE International Conference on Cyber Security and Resilience (CSR), 2021.","DOI":"10.1109\/CSR51186.2021.9527993"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_052","doi-asserted-by":"crossref","unstructured":"M. Khanna, A. Chaudhary, A. Toofani, and A. Pawar, \u201cPerformance comparison of multi-objective algorithms for test case prioritization during web application testing,\u201d AJSE, vol. 44, no. 11, pp. 9599\u20139625, 2019. https:\/\/doi.org\/10.1007\/s13369-019-03817-7.","DOI":"10.1007\/s13369-019-03817-7"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_053","doi-asserted-by":"crossref","unstructured":"M. Khanna, N. Chauhan, D. Sharma, and A. Toofani, \u201cA novel approach for regression testing of web applications,\u201d Int. J. Intell. Syst. Appl., vol. 10, no. 2, pp. 55\u201371, 2018. https:\/\/doi.org\/10.5815\/ijisa.2018.02.06.","DOI":"10.5815\/ijisa.2018.02.06"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_028","unstructured":"C. Murphy, G. E. Kaiser, and L. Hu, \u201cProperties of machine learning applications for use in metamorphic testing,\u201d in Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE \u201908), vol. 8, pp. 867\u2013872, 2008."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_034","doi-asserted-by":"crossref","unstructured":"B. V. Ramana, M. S. P. Babu, and N. Venkateswarlu, \u201cA critical study of selected classification algorithms for liver disease diagnosis,\u201d Int. J.\u00a0Database Manag. Syst., vol.\u00a03, no. 2, pp.\u00a0101\u2013114, 2011.","DOI":"10.5121\/ijdms.2011.3207"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_039","doi-asserted-by":"crossref","unstructured":"S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cort\u00e9s, \u201cA survey on metamorphic testing,\u201d IEEE Trans. Software Eng., vol.\u00a042, no. 9, pp.\u00a0805\u2013824, 2016, https:\/\/doi.org\/10.1109\/tse.2016.2532875.","DOI":"10.1109\/TSE.2016.2532875"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_044","unstructured":"I. H. Witten, E. Frank, M. A. Hall, C. J. Pal, and M. Data, \u201cPractical machine learning tools and techniques,\u201d Paper presented at the Data mining, 2005."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_020","doi-asserted-by":"crossref","unstructured":"S. Kumar, \u201cReviewing software testing models and optimization techniques: an analysis of efficiency and advancement needs,\u201d J.\u00a0Comput. Mech. Manag., vol.\u00a02, no. 1, pp.\u00a043\u201355, 2023, https:\/\/doi.org\/10.57159\/gadl.jcmm.2.1.23041.","DOI":"10.57159\/gadl.jcmm.2.1.23041"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_042","doi-asserted-by":"crossref","unstructured":"F. Tambon, F. Khomh, G. J.\u00a0I. Antoniol, and S. Technology, \u201cA probabilistic framework for mutation testing in deep neural networks,\u201d Inf. Software Technol., vol. 155, no. 2, p. 107129, 2023. https:\/\/doi.org\/10.1016\/j.infsof.2022.107129.","DOI":"10.1016\/j.infsof.2022.107129"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_005","doi-asserted-by":"crossref","unstructured":"J.\u00a0H. Bernardo, D. A. Da Costa, S. Q. D. Medeiros, and U. Kulesza, \u201cHow do machine learning projects use continuous integration practices? An empirical study on GitHub actions,\u201d Paper presented at the Proceedings of the 21st International Conference on Mining Software Repositories, 2024.","DOI":"10.1145\/3643991.3644915"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_019","doi-asserted-by":"crossref","unstructured":"M. Kim, S. Sinha, and A. Orso, \u201cAdaptive REST API testing with reinforcement learning,\u201d Paper presented at the 2023\u00a038th IEEE\/ACM International Conference on Automated Software Engineering (ASE), 2023.","DOI":"10.1109\/ASE56229.2023.00218"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_022","doi-asserted-by":"crossref","unstructured":"Y. Liu, R. Li, H. Tao, and Z. Zheng, \u201cTest adequacy criteria for metamorphic testing,\u201d Paper presented at the 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C), 2024.","DOI":"10.1109\/QRS-C63300.2024.00072"},{"key":"2026021619094514479_j_jisys-2024-0363_ref_037","unstructured":"P. Saha, Improving the Effectiveness of Metamorphic Testing Using Systematic Test Case Generation, Bozeman, Montana State University, 2024."},{"key":"2026021619094514479_j_jisys-2024-0363_ref_008","unstructured":"W. R. Shadish, T. D. Cook, and D. T. Campbell, Experimental and quasi-experimental designs for generalized causal inference, Boston, New York, Houghton Mifflin, 2007. pp. 623."}],"container-title":["Journal of Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2024-0363\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2024-0363\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T19:10:00Z","timestamp":1771269000000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2024-0363\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,1]]},"references-count":53,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,2,16]]},"published-print":{"date-parts":[[2026,1,23]]}},"alternative-id":["10.1515\/jisys-2024-0363"],"URL":"https:\/\/doi.org\/10.1515\/jisys-2024-0363","relation":{},"ISSN":["2191-026X"],"issn-type":[{"value":"2191-026X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,1]]},"article-number":"20240363"}}