{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T16:00:30Z","timestamp":1781366430425,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":35,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,18]],"date-time":"2021-08-18T00:00:00Z","timestamp":1629244800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001031","name":"Australian Respiratory Council","doi-asserted-by":"publisher","award":["DP210102447"],"award-info":[{"award-number":["DP210102447"]}],"id":[{"id":"10.13039\/501100001031","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,20]]},"DOI":"10.1145\/3468264.3473136","type":"proceedings-article","created":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T01:40:20Z","timestamp":1629337220000},"page":"1487-1490","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["New visions on metamorphic testing after a quarter of a century of inception"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3578-0994","authenticated-orcid":false,"given":"Tsong Yueh","family":"Chen","sequence":"first","affiliation":[{"name":"Swinburne University of Technology, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0460-8377","authenticated-orcid":false,"given":"T. H.","family":"Tse","sequence":"additional","affiliation":[{"name":"University of Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,8,18]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00023"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3072166"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_1_4_1","unstructured":"T. Y. Chen P. L. Poon K. Qiu Z. Zheng and J. Zhou. 2021. Use of metamorphic relations as knowledge carriers to train deep neural networks. arXiv preprint arXiv:2104.04718.  T. Y. Chen P. L. Poon K. Qiu Z. Zheng and J. Zhou. 2021. Use of metamorphic relations as knowledge carriers to train deep neural networks. arXiv preprint arXiv:2104.04718."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.23"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408314"},{"key":"e_1_3_2_1_7_1","unstructured":"P. Dockrill. 2019. Study claims one punctuation mark has been skewing our scientific ranking system. ScienceAlert. Retrieved from https:\/\/www.sciencealert.com\/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals.  P. Dockrill. 2019. Study claims one punctuation mark has been skewing our scientific ranking system. ScienceAlert. Retrieved from https:\/\/www.sciencealert.com\/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals."},{"key":"e_1_3_2_1_8_1","volume-title":"Proc. ACM Program. Lang. 1(OOPSLA), 93:1\u201393:29","author":"Donaldson A. F.","unstructured":"A. F. Donaldson , H. Evrard , A. Lascu , and P. Thomson . 2017. Automated testing of graphics shader compilers . Proc. ACM Program. Lang. 1(OOPSLA), 93:1\u201393:29 . A. F. Donaldson, H. Evrard, A. Lascu, and P. Thomson. 2017. Automated testing of graphics shader compilers. Proc. ACM Program. Lang. 1(OOPSLA), 93:1\u201393:29."},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201918)","author":"Dwarakanath A.","unstructured":"A. Dwarakanath , M. Ahuja , S. Sikand , R. M. Rao , R. P. J. C. Bose , N. Dubash , and S. Podder . 2018. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing . Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201918) , 118\u2013128. A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. P. J. C. Bose, N. Dubash, and S. Podder. 2018. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201918), 118\u2013128."},{"key":"e_1_3_2_1_10_1","unstructured":"Google-GraphicsFuzz. 2018. GraphicsFuzz is acquired by Google. Retrieved from https:\/\/www.graphicsfuzz.com\/.  Google-GraphicsFuzz. 2018. GraphicsFuzz is acquired by Google. Retrieved from https:\/\/www.graphicsfuzz.com\/."},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920)","author":"He P.","unstructured":"P. He , C. Meister , and Z. Su . 2020. Structure-invariant testing for machine translation . Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920) , 961\u2013973. P. He, C. Meister, and Z. Su. 2020. Structure-invariant testing for machine translation. Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920), 961\u2013973."},{"key":"e_1_3_2_1_12_1","volume-title":"Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201920)","author":"He X.","unstructured":"X. He , X. Wang , J. Shi , and Y. Liu . 2020. Testing high performance numerical simulation programs: Experience, lessons learned, and open issues . Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201920) , 502\u2013515. X. He, X. Wang, J. Shi, and Y. Liu. 2020. Testing high performance numerical simulation programs: Experience, lessons learned, and open issues. Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201920), 502\u2013515."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.233816"},{"key":"e_1_3_2_1_14_1","volume-title":"Proceedings of the 2nd International Workshop on Metamorphic Testing (MET\u201917)","author":"Jarman D. C.","unstructured":"D. C. Jarman , Z. Q. Zhou , and T. Y. Chen . 2017. Metamorphic testing for Adobe data analytics software . Proceedings of the 2nd International Workshop on Metamorphic Testing (MET\u201917) , 21\u201327. D. C. Jarman, Z. Q. Zhou, and T. Y. Chen. 2017. Metamorphic testing for Adobe data analytics software. Proceedings of the 2nd International Workshop on Metamorphic Testing (MET\u201917), 21\u201327."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.04.002"},{"key":"e_1_3_2_1_16_1","volume-title":"Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201914)","author":"Le V.","unstructured":"V. Le , M. Afshari , and Z. Su . 2014. Compiler validation via equivalence modulo inputs . Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201914) , 216\u2013226. V. Le, M. Afshari, and Z. Su. 2014. Compiler validation via equivalence modulo inputs. Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201914), 216\u2013226."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3029468"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.348"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3009698"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912)","author":"Ramanathan A.","unstructured":"A. Ramanathan , C. A. Steed , and L. L. Pullum . 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics . Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912) , 68\u201373. A. Ramanathan, C. A. Steed, and L. L. Pullum. 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912), 68\u201373."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_3_2_1_24_1","volume-title":"A vocabulary of program slicing-based techniques. ACM Comp. Surv., 44(3)","author":"Silva J.","year":"2012","unstructured":"J. Silva . 2012. A vocabulary of program slicing-based techniques. ACM Comp. Surv., 44(3) ( 2012 ), 12:1\u201312:41. J. Silva. 2012. A vocabulary of program slicing-based techniques. ACM Comp. Surv., 44(3) (2012), 12:1\u201312:41."},{"key":"e_1_3_2_1_25_1","volume-title":"Proceedings of the 40th International Conference on Software Engineering (ICSE\u201918)","author":"Tian Y.","unstructured":"Y. Tian , K. Pei , S. Jana , and B. Ray . 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars . Proceedings of the 40th International Conference on Software Engineering (ICSE\u201918) , 303\u2013314. Y. Tian, K. Pei, S. Jana, and B. Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. Proceedings of the 40th International Conference on Software Engineering (ICSE\u201918), 303\u2013314."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010248"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"X. Xie W. E. Wong T. Y. Chen and B. Xu. 2013. Metamorphic slice: An application in spectrum-based fault localization. Info. Soft. Tech. 55(5) (2013) 866\u2013879.  X. Xie W. E. Wong T. Y. Chen and B. Xu. 2013. Metamorphic slice: An application in spectrum-based fault localization. Info. Soft. Tech. 55(5) (2013) 866\u2013879.","DOI":"10.1016\/j.infsof.2012.08.008"},{"key":"e_1_3_2_1_29_1","volume-title":"METTLE: A metamorphic testing approach to assessing and validating unsupervised machine learning systems","author":"Xie X.","year":"2020","unstructured":"X. Xie , Z. Zhang , T. Y. Chen , Y. Liu , P. L. Poon , and B. Xu . 2020 . METTLE: A metamorphic testing approach to assessing and validating unsupervised machine learning systems . IEEE Trans. Reliab ., 69(4) (2020), 1293\u20131322. X. Xie, Z. Zhang, T. Y. Chen, Y. Liu, P. L. Poon, and B. Xu. 2020. METTLE: A metamorphic testing approach to assessing and validating unsupervised machine learning systems. IEEE Trans. Reliab., 69(4) (2020), 1293\u20131322."},{"key":"e_1_3_2_1_30_1","volume-title":"Proceedings of 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201919)","author":"Zhang B.","unstructured":"B. Zhang , H. Zhang , J. Chen , D. Hao , and P. Moscato . 2019. Automatic discovery and cleansing of numerical metamorphic relations . Proceedings of 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201919) , 235\u2013245. B. Zhang, H. Zhang, J. Chen, D. Hao, and P. Moscato. 2019. Automatic discovery and cleansing of numerical metamorphic relations. Proceedings of 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201919), 235\u2013245."},{"key":"e_1_3_2_1_31_1","article-title":"2020. Machine learning testing: Survey, landscapes and horizons","author":"Zhang J. M.","unstructured":"J. M. Zhang , M. Harman , L. Ma , and Y. Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE Trans. Soft. Eng. , doi:10.1109\/TSE.2019.2962027. J. M. Zhang, M. Harman, L. Ma, and Y. Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Soft. Eng., doi:10.1109\/TSE.2019.2962027.","journal-title":"IEEE Trans. Soft. Eng."},{"key":"e_1_3_2_1_32_1","volume-title":"Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE\u201918)","author":"Zhang M.","unstructured":"M. Zhang , Y. Zhang , L. Zhang , C. Liu , and S. Khurshid . 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems . Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE\u201918) , 132\u2013142. M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE\u201918), 132\u2013142."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3241979"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876433"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2915065"}],"event":{"name":"ESEC\/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Athens Greece","acronym":"ESEC\/FSE '21","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3473136","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3468264.3473136","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:22Z","timestamp":1750191442000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3473136"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,18]]},"references-count":35,"alternative-id":["10.1145\/3468264.3473136","10.1145\/3468264"],"URL":"https:\/\/doi.org\/10.1145\/3468264.3473136","relation":{},"subject":[],"published":{"date-parts":[[2021,8,18]]},"assertion":[{"value":"2021-08-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}