{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T14:53:21Z","timestamp":1784904801606,"version":"3.55.0"},"reference-count":97,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T00:00:00Z","timestamp":1515024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"General Research Fund of the Research Grants Council of Hong Kong","award":["716612"],"award-info":[{"award-number":["716612"]}]},{"name":"International Doctoral Innovation Centre"},{"DOI":"10.13039\/501100000837","name":"University of Nottingham","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000837","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100007928","name":"Ningbo Science and Technology Bureau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007928","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["LP160101691"],"award-info":[{"award-number":["LP160101691"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Artificial Intelligence and Optimisation Research Group of University of Nottingham Ningbo China"},{"DOI":"10.13039\/501100007929","name":"Ningbo Education Bureau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007929","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.<\/jats:p>","DOI":"10.1145\/3143561","type":"journal-article","created":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T16:27:31Z","timestamp":1515083251000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":420,"title":["Metamorphic Testing"],"prefix":"10.1145","volume":"51","author":[{"given":"Tsong Yueh","family":"Chen","sequence":"first","affiliation":[{"name":"Swinburne University of Technology, Hawthorn, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei-Ching","family":"Kuo","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Hawthorn, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3125-4399","authenticated-orcid":false,"given":"Huai","family":"Liu","sequence":"additional","affiliation":[{"name":"Victoria University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pak-Lok","family":"Poon","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dave","family":"Towey","sequence":"additional","affiliation":[{"name":"University of Nottingham Ningbo China, Ningbo, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"T. H.","family":"Tse","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Pokfulam, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhi Quan","family":"Zhou","sequence":"additional","affiliation":[{"name":"University of Wollongong, Wollongong, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,1,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479440.2482677"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/12.2185"},{"key":"e_1_2_1_3_1","volume-title":"ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing","volume":"249","author":"Aruna Chittineni","unstructured":"Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies . In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing , Vol. 249 . Springer, Berlin, 399--410. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2016.2547380"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916)","author":"Barus Arlinta Christy","unstructured":"Arlinta Christy Barus , Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu , and Heinz W. Schmidt . 2016. The impact of source test case selection on the effectiveness of metamorphic testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916) . ACM, New York, NY, 5--11. Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Heinz W. Schmidt. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916). ACM, New York, NY, 5--11."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2006.161"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.1984.233702"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2008.12.001"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408795"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2013.43"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194006002951"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.4018\/jwsr.2007040103"},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24.  Tsong Yueh Chen Joshua W. K. Ho Huai Liu and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10 article no. 24.","DOI":"10.1186\/1471-2105-10-24"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC\u201904)","author":"Chen Tsong Yueh","year":"2004","unstructured":"Tsong Yueh Chen , De Hao Huang , T. H. Tse , and Zhi Quan Zhou . 2004 . Case studies on the selection of useful relations in metamorphic testing . In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC\u201904) . Polytechnic University of Madrid, Madrid, Spain, 569--583. Tsong Yueh Chen, De Hao Huang, T. H. Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC\u201904). Polytechnic University of Madrid, Madrid, Spain, 569--583."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02138-1_19"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel\/Distributed Computing (SNPD\u201904)","author":"Chen Tsong Yueh","year":"2004","unstructured":"Tsong Yueh Chen , Fei-Ching Kuo , Ying Liu , and Antony Tang . 2004 . Metamorphic testing and testing with special values . In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel\/Distributed Computing (SNPD\u201904) . International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134. Tsong Yueh Chen, Fei-Ching Kuo, Ying Liu, and Antony Tang. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel\/Distributed Computing (SNPD\u201904). International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2016.176"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.02.022"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9.  Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9.","DOI":"10.1007\/s11432-015-5314-x"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1083231.1083236"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2003.1214323"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.07.037"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(01)00028-0"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/566172.566202"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(02)00129-5"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.23"},{"key":"e_1_2_1_30_1","volume-title":"Advances in Computers","author":"Cohen David","unstructured":"David Cohen , Mikael Lindvall , and Patricia Costa . 2004. An introduction to agile methods . In Advances in Computers , Vol. 62 . Elsevier, Amsterdam , The Netherlands , 1--66. David Cohen, Mikael Lindvall, and Patricia Costa. 2004. An introduction to agile methods. In Advances in Computers, Vol. 62. Elsevier, Amsterdam, The Netherlands, 1--66."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/355604.361591"},{"key":"e_1_2_1_32_1","volume-title":"A machine learning based framework for verification and validation of massive scale image data","author":"Ding Junhua","unstructured":"Junhua Ding , Xin-Hua Hu , and Venkat Gudivada . 2017. A machine learning based framework for verification and validation of massive scale image data . IEEE Transactions on Big Data . DOI:10.1109\/TBDATA.2017.2680460. 10.1109\/TBDATA.2017.2680460 Junhua Ding, Xin-Hua Hu, and Venkat Gudivada. 2017. A machine learning based framework for verification and validation of massive scale image data. IEEE Transactions on Big Data. DOI:10.1109\/TBDATA.2017.2680460."},{"key":"e_1_2_1_33_1","first-page":"437","article-title":"Case studies on testing with compositional metamorphic relations","volume":"24","author":"Dong Guowei","year":"2008","unstructured":"Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2008 . Case studies on testing with compositional metamorphic relations . Journal of Southeast University (English Edition) 24 , 4, 437 -- 443 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.","journal-title":"Journal of Southeast University (English Edition)"},{"key":"e_1_2_1_34_1","first-page":"130","article-title":"Survey of metamorphic testing","volume":"3","author":"Dong Guowei","year":"2009","unstructured":"Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2009 . Survey of metamorphic testing . Journal of Frontiers of Computer Science and Technology 3 , 2, 130 -- 143 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.","journal-title":"Journal of Frontiers of Computer Science and Technology"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1181775.1181806"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/359488.359495"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.2003.1245319"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2007.4385527"},{"key":"e_1_2_1_39_1","volume-title":"Encyclopedia of Software Engineering","author":"Hamlet Richard","unstructured":"Richard Hamlet . 2002. Random testing . In Encyclopedia of Software Engineering . John Wiley , New York, NY . Richard Hamlet. 2002. Random testing. In Encyclopedia of Software Engineering. John Wiley, New York, NY."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102580"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA\u201906)","author":"Hu Peifeng","unstructured":"Peifeng Hu , Zhenyu Zhang , Wing Kwong Chan , and T. H. Tse . 2006. An empirical comparison between direct and indirect test result checking approaches . In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA\u201906) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT\u201906\/FSE-14). ACM, New York, NY, 6--13. Peifeng Hu, Zhenyu Zhang, Wing Kwong Chan, and T. H. Tse. 2006. An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA\u201906) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT\u201906\/FSE-14). ACM, New York, NY, 6--13."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCSE.2013.16"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/257734.257766"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS\u201913)","author":"Jiang Mingyue","year":"2013","unstructured":"Mingyue Jiang , Tsong Yueh Chen , Fei-Ching Kuo , and Zuohua Ding . 2013 . Testing central processing unit scheduling algorithms using metamorphic testing . In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS\u201913) . IEEE Computer Society, Los Alamitos, CA, 530--536. Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS\u201913). IEEE Computer Society, Los Alamitos, CA, 530--536."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.04.002"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2015.79"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1594"},{"key":"e_1_2_1_48_1","volume-title":"Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck.","author":"Ko Andrew J.","year":"2011","unstructured":"Andrew J. Ko , Robin Abraham , Laura Beckwith , Alan Blackwell , Margaret Burnett , Martin Erwig , Chris Scaffidi , Joseph Lawrance , Henry Lieberman , Brad Myers , Mary Beth Rosson , Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011 . The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44. Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/LCN.2011.6115306"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201915)","author":"Lidbury Christopher","unstructured":"Christopher Lidbury , Andrei Lascu , Nathan Chong , and Alastair F. Donaldson . 2015. Many-core compiler fuzzing . In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201915) . ACM, New York, NY, 65--76. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI\u201915). ACM, New York, NY, 65--76."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915)","volume":"2","author":"Lindvall Mikael","unstructured":"Mikael Lindvall , Dharmalingam Ganesan , Ragnar \u00c1rdal , and Robert E. Wiegand . 2015. Metamorphic model-based testing applied on NASA DAT: An experience report . In Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915) , Vol. 2 . IEEE, Piscataway, NJ, 129--138. Mikael Lindvall, Dharmalingam Ganesan, Ragnar \u00c1rdal, and Robert E. Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT: An experience report. In Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915), Vol. 2. IEEE, Piscataway, NJ, 129--138."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916)","author":"Lindvall Mikael","unstructured":"Mikael Lindvall , Dharmalingam Ganesan , Sigurthor Bjorgvinsson , Kristjan Jonsson , Haukur Steinn Logason , Frederik Dietrich , and Robert E. Wiegand . 2016. Agile metamorphic model-based testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916) . ACM, New York, NY, 26--32. Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. 2016. Agile metamorphic model-based testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916). ACM, New York, NY, 26--32."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET.2010.28"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2012.10"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591109"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT\u201906\/FSE-14)","author":"Lu Heng","unstructured":"Heng Lu , Wing Kwong Chan , and T. H. Tse . 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation . In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT\u201906\/FSE-14) . ACM, New York, NY, 242--252. Heng Lu, Wing Kwong Chan, and T. H. Tse. 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT\u201906\/FSE-14). ACM, New York, NY, 242--252."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/UIC-ATC.2010.75"},{"key":"e_1_2_1_60_1","volume-title":"On the Effectiveness of Random Testing. Master\u2019s Thesis. Department of Computer Science","author":"Mak Ieng Kei","unstructured":"Ieng Kei Mak . 1997. On the Effectiveness of Random Testing. Master\u2019s Thesis. Department of Computer Science , University of Melbourne , Melbourne, Australia . Ieng Kei Mak. 1997. On the Effectiveness of Random Testing. Master\u2019s Thesis. Department of Computer Science, University of Melbourne, Melbourne, Australia."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.409"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2006.24"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE\u201908)","author":"Murphy Christian","year":"2008","unstructured":"Christian Murphy , Gail Kaiser , Lifeng Hu , and Leon Wu . 2008 . Properties of machine learning applications for use in metamorphic testing . In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE\u201908) . Knowledge Systems Institute Graduate School, Skokie, IL, 867--872. Christian Murphy, Gail Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of machine learning applications for use in metamorphic testing. In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE\u201908). Knowledge Systems Institute Graduate School, Skokie, IL, 867--872."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/1987993.1988003"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2016.049"},{"key":"e_1_2_1_68_1","volume-title":"Nardi","author":"Oliveira Rafael A. P.","year":"2015","unstructured":"Rafael A. P. Oliveira , Upulee Kanewala , and Paulo A . Nardi . 2015 . Automated test oracles: State of the art, taxonomies, and trends. In Advances in Computers, Vol. 95 . Elsevier , Amsterdam, The Netherlands, 113--199. Rafael A. P. Oliveira, Upulee Kanewala, and Paulo A. Nardi. 2015. Automated test oracles: State of the art, taxonomies, and trends. In Advances in Computers, Vol. 95. Elsevier, Amsterdam, The Netherlands, 113--199."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593885"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2014.76"},{"key":"e_1_2_1_71_1","unstructured":"Krishna Patel and Robert M. Hierons. submitted for publication. A systematic literature review on testing and debugging non-testable systems. Retrieved from http:\/\/people.brunel.ac.uk\/&sim;csstrmh\/Intt\/synth.pdf.  Krishna Patel and Robert M. Hierons. submitted for publication. A systematic literature review on testing and debugging non-testable systems. Retrieved from http:\/\/people.brunel.ac.uk\/&sim;csstrmh\/Intt\/synth.pdf."},{"key":"e_1_2_1_72_1","volume-title":"Advances in Computers","author":"Pezz\u00e8 Mauro","unstructured":"Mauro Pezz\u00e8 and Cheng Zhang . 2014. Automated test oracles: A survey . In Advances in Computers , Vol. 95 . Academic Press , Waltham, MA , 1--48. Mauro Pezz\u00e8 and Cheng Zhang. 2014. Automated test oracles: A survey. In Advances in Computers, Vol. 95. Academic Press, Waltham, MA, 1--48."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1108\/ITP-01-2013-0004"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.4018\/JOEUC.2017040102"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912)","author":"Laura","unstructured":"Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models . In Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912) . IEEE Computer Society, Los Alamitos, CA, 62--67. Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models. In Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912). IEEE Computer Society, Los Alamitos, CA, 62--67."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912)","author":"Ramanathan Arvind","unstructured":"Arvind Ramanathan , Chad A. Steed , and Laura L. Pullum . 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics . In Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912) . IEEE Computer Society, Los Alamitos, CA, 68--73. Arvind Ramanathan, Chad A. Steed, and Laura L. Pullum. 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. In Proceedings of the 2012 ASE\/IEEE International Conference on BioMedical Computing (BioMedCom\u201912). IEEE Computer Society, Los Alamitos, CA, 68--73."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2013.18"},{"key":"e_1_2_1_78_1","unstructured":"John Regehr. 2014. Finding compiler bugs by removing dead code. Retrieved from http:\/\/blog.regehr.org\/archives\/1161.  John Regehr. 2014. Finding compiler bugs by removing dead code. Retrieved from http:\/\/blog.regehr.org\/archives\/1161."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.341844"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1142\/S021972001100563X"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2010.20"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.11.002"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2011.65"},{"key":"e_1_2_1_85_1","unstructured":"Systems and software engineering: Systems and software Quality Requirements and Evaluation (SQuaRE): System and software quality models. ISO\/IEC 25010:2011 ISO.  Systems and software engineering: Systems and software Quality Requirements and Evaluation (SQuaRE): System and software quality models. ISO\/IEC 25010:2011 ISO."},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2010.39"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/TALE.2015.7386036"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/TALE.2016.7851797"},{"key":"e_1_2_1_89_1","volume-title":"Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916)","author":"Troup Michael","unstructured":"Michael Troup , Andrian Yang , Amir Hossein Kamali , Eleni Giannoulatou , Tsong Yueh Chen , and Joshua W. K. Ho . 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916) . ACM, New York, NY, 33--36. Michael Troup, Andrian Yang, Amir Hossein Kamali, Eleni Giannoulatou, Tsong Yueh Chen, and Joshua W. K. Ho. 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916). ACM, New York, NY, 33--36."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.2004.1342879"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2011.20"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.08.008"},{"key":"e_1_2_1_94_1","first-page":"1","article-title":"A brief survey of program slicing","volume":"30","author":"Xu Baowen","year":"2005","unstructured":"Baowen Xu , Ju Qian , Xiaofang Zhang , Zhongqiang Wu , and Lin Chen . 2005 . A brief survey of program slicing . ACM SIGSOFT Software Engineering Notes 30 , 2, 1 -- 36 . Baowen Xu, Ju Qian, Xiaofang Zhang, Zhongqiang Wu, and Lin Chen. 2005. A brief survey of program slicing. ACM SIGSOFT Software Engineering Notes 30, 2, 1--36.","journal-title":"ACM SIGSOFT Software Engineering Notes"},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915)","volume":"2","author":"Yusuf Iman I.","unstructured":"Iman I. Yusuf , Ian E. Thomas , Maria Spichkova , Steve Androulakis , Grischa R. Meyer , Daniel W. Drumm , George Opletal , Salvy P. Russo , Ashley M. Buckle , and Heinz W. Schmidt . 2015. Chiminey: Reliable computing and data management platform in the cloud . In Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915) , Vol. 2 . IEEE, Piscataway, NJ, 677--680. Iman I. Yusuf, Ian E. Thomas, Maria Spichkova, Steve Androulakis, Grischa R. Meyer, Daniel W. Drumm, George Opletal, Salvy P. Russo, Ashley M. Buckle, and Heinz W. Schmidt. 2015. Chiminey: Reliable computing and data management platform in the cloud. In Proceedings of the 37th International Conference on Software Engineering (ICSE\u201915), Vol. 2. IEEE, Piscataway, NJ, 677--680."},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.3724\/SP.J.1001.2009.00578"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2478001"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.437"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSA.2015.13"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3143561","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3143561","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:21Z","timestamp":1750212801000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3143561"}},"subtitle":["A Review of Challenges and Opportunities"],"short-title":[],"issued":{"date-parts":[[2018,1,4]]},"references-count":97,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3143561"],"URL":"https:\/\/doi.org\/10.1145\/3143561","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,4]]},"assertion":[{"value":"2017-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}