{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T01:49:33Z","timestamp":1769737773113,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":25,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T00:00:00Z","timestamp":1603152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,20]]},"DOI":"10.1145\/3425174.3425226","type":"proceedings-article","created":{"date-parts":[[2020,10,23]],"date-time":"2020-10-23T00:01:25Z","timestamp":1603411285000},"page":"98-106","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["An Experimental Study on Applying Metamorphic Testing in Machine Learning Applications"],"prefix":"10.1145","author":[{"given":"Sebasti\u00e3o H. N.","family":"Santos","sequence":"first","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o S\u00e3o, Carlos, SP"}]},{"given":"Beatriz Nogueira Carvalho","family":"da Silveira","sequence":"additional","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o S\u00e3o, Carlos, SP"}]},{"given":"Stev\u00e3o A.","family":"Andrade","sequence":"additional","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o S\u00e3o, Carlos, SP"}]},{"given":"M\u00e1rcio","family":"Delamaro","sequence":"additional","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o S\u00e3o, Carlos, SP"}]},{"given":"Simone R. S.","family":"Souza","sequence":"additional","affiliation":[{"name":"Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o S\u00e3o, Carlos, SP"}]}],"member":"320","published-online":{"date-parts":[[2020,10,22]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2016.7535456"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Sylvain Arlot Alain Celisse etal 2010. A survey of cross-validation procedures for model selection. Statistics surveys 4 (2010) 40--79.  Sylvain Arlot Alain Celisse et al. 2010. A survey of cross-validation procedures for model selection. Statistics surveys 4 (2010) 40--79.","DOI":"10.1214\/09-SS054"},{"key":"e_1_3_2_1_3_1","volume-title":"The goal question metric approach. Encyclopedia of software engineering","author":"Gianluigi Caldiera Victor R","year":"1994","unstructured":"Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach . 1994. The goal question metric approach. Encyclopedia of software engineering ( 1994 ), 528--532. Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528--532."},{"key":"e_1_3_2_1_4_1","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases. arXiv:2002.12543 [cs.SE]","author":"Chen T. Y.","year":"1998","unstructured":"T. Y. Chen , S. C. Cheung , and S. M. Yiu . 1998 . Metamorphic Testing: A New Approach for Generating Next Test Cases. arXiv:2002.12543 [cs.SE] T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. arXiv:2002.12543 [cs.SE]"},{"key":"e_1_3_2_1_5_1","volume-title":"Huai Liu, and Xiaoyuan Xie.","author":"Chen Tsong Yueh","year":"2009","unstructured":"Tsong Yueh Chen , Joshua WK Ho , Huai Liu, and Xiaoyuan Xie. 2009 . An innovative approach for testing bioinformatics programs using metamorphic testing. BMC bioinformatics 10, 1 (2009), 24. Tsong Yueh Chen, Joshua WK Ho, Huai Liu, and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC bioinformatics 10, 1 (2009), 24."},{"key":"e_1_3_2_1_6_1","unstructured":"Gregory W Corder and Dale I Foreman. 2011. Nonparametric statistics for non-statisticians.  Gregory W Corder and Dale I Foreman. 2011. Nonparametric statistics for non-statisticians."},{"key":"e_1_3_2_1_7_1","unstructured":"Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml  Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_3_2_1_9_1","volume-title":"Potential biases in machine learning algorithms using electronic health record data. JAMA internal medicine 178, 11","author":"Gianfrancesco Milena A","year":"2018","unstructured":"Milena A Gianfrancesco , Suzanne Tamang , Jinoos Yazdany , and Gabriela Schmajuk . 2018. Potential biases in machine learning algorithms using electronic health record data. JAMA internal medicine 178, 11 ( 2018 ), 1544--1547. Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. 2018. Potential biases in machine learning algorithms using electronic health record data. JAMA internal medicine 178, 11 (2018), 1544--1547."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.18.20224"},{"key":"e_1_3_2_1_11_1","volume-title":"Brian Mac Namee, and Aoife D'arcy","author":"Kelleher John D","year":"2015","unstructured":"John D Kelleher , Brian Mac Namee, and Aoife D'arcy . 2015 . Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies. MIT press , Cambridge, United Kingdom. John D Kelleher, Brian Mac Namee, and Aoife D'arcy. 2015. Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies. MIT press, Cambridge, United Kingdom."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2014.11.005"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2016.51"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.21917\/ijsc.2015.0133"},{"key":"e_1_3_2_1_15_1","volume-title":"Dataset Coverage for Testing Machine Learning Computer Programs. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE Computer Society","author":"Nakajima S.","year":"2016","unstructured":"S. Nakajima and H. Bui . 2016 . Dataset Coverage for Testing Machine Learning Computer Programs. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE Computer Society , Los Alamitos, CA, USA, 297--304. https:\/\/doi.org\/10.1109\/APSEC. 2016 .049 10.1109\/APSEC.2016.049 S. Nakajima and H. Bui. 2016. Dataset Coverage for Testing Machine Learning Computer Programs. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE Computer Society, Los Alamitos, CA, USA, 297--304. https:\/\/doi.org\/10.1109\/APSEC.2016.049"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2007.01.006"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/25.4.465"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-4625-2"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_2_1_23_1","volume-title":"Yang Liu, Pak-Lok Poon, and Baowen Xu.","author":"Xie Xiaoyuan","year":"2020","unstructured":"Xiaoyuan Xie , Zhiyi Zhang , Tsong Yueh Chen , Yang Liu, Pak-Lok Poon, and Baowen Xu. 2020 . METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems. IEEE Transactions on Reliability ( 2020), 1--30. Xiaoyuan Xie, Zhiyi Zhang, Tsong Yueh Chen, Yang Liu, Pak-Lok Poon, and Baowen Xu. 2020. METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems. IEEE Transactions on Reliability (2020), 1--30."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the 8th International Symposium on Future Software Technology (ISFST","author":"Zhou Zhi Quan","year":"2004","unstructured":"Zhi Quan Zhou , DH Huang , TH Tse , Zongyuan Yang , Haitao Huang , and TY Chen . 2004 . Metamorphic testing and its applications . In Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004). Software Engineers Association Xian, China, Citeseerx, Xian, China, 346--351. Zhi Quan Zhou, DH Huang, TH Tse, Zongyuan Yang, Haitao Huang, and TY Chen. 2004. Metamorphic testing and its applications. In Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004). Software Engineers Association Xian, China, Citeseerx, Xian, China, 346--351."}],"event":{"name":"SAST 20: 5th Brazilian Symposium on Systematic and Automated Software Testing","location":"Natal Brazil","acronym":"SAST 20","sponsor":["SBC Brazilian Computer Society"]},"container-title":["Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3425174.3425226","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3425174.3425226","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:54Z","timestamp":1750195914000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3425174.3425226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,20]]},"references-count":25,"alternative-id":["10.1145\/3425174.3425226","10.1145\/3425174"],"URL":"https:\/\/doi.org\/10.1145\/3425174.3425226","relation":{},"subject":[],"published":{"date-parts":[[2020,10,20]]},"assertion":[{"value":"2020-10-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}