{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T13:30:55Z","timestamp":1770557455502,"version":"3.49.0"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,5,27]],"date-time":"2025-05-27T00:00:00Z","timestamp":1748304000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council Project","award":["DP210102447"],"award-info":[{"award-number":["DP210102447"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62272037"],"award-info":[{"award-number":["62272037"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>Metamorphic testing has become one mainstream technique to address the notorious oracle problem in software testing, thanks to its great successes in revealing real-life bugs in a wide variety of software systems. Metamorphic relations, the core component of metamorphic testing, have continuously attracted research interests from both academia and industry. In the last decade, a rapidly increasing number of studies have been conducted to systematically generate metamorphic relations from various sources and for different application domains. In this article, based on the systematic review on the state of the art for metamorphic relations\u2019 generation, we summarize and highlight visions for further advancing the theory and techniques for identifying and constructing metamorphic relations and discuss promising research directions in related areas.<\/jats:p>","DOI":"10.1145\/3708521","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T12:05:27Z","timestamp":1734523527000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Metamorphic Relation Generation: State of the Art and Research Directions"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7669-4112","authenticated-orcid":false,"given":"Rui","family":"Li","sequence":"first","affiliation":[{"name":"Deakin University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3125-4399","authenticated-orcid":false,"given":"Huai","family":"Liu","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2840-2418","authenticated-orcid":false,"given":"Pak-Lok","family":"Poon","sequence":"additional","affiliation":[{"name":"Central Queensland University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0877-4353","authenticated-orcid":false,"given":"Dave","family":"Towey","sequence":"additional","affiliation":[{"name":"University of Nottingham Ningbo China, Ningbo, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3696-6176","authenticated-orcid":false,"given":"Chang-Ai","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7922-9067","authenticated-orcid":false,"given":"Zheng","family":"Zheng","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering, Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1769-9367","authenticated-orcid":false,"given":"Zhi Quan","family":"Zhou","sequence":"additional","affiliation":[{"name":"University of Wollongong, Wollongong, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3578-0994","authenticated-orcid":false,"given":"Tsong Yueh","family":"Chen","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2025,5,27]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896971.2896980"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473920"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3407840"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896971.2896977"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111041"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3256322"},{"key":"e_1_3_1_9_2","first-page":"191","volume-title":"Proceedings of the IASTED International Conference on Software Engineering (SE \u201998)","author":"Chan F. T.","year":"1998","unstructured":"F. T. Chan, Tsong Yueh Chen, Shing-Chi Cheung, Man Fai Lau, and Siu-Ming Yiu. 1998. Application of metamorphic testing in numerical analysis. In Proceedings of the IASTED International Conference on Software Engineering (SE \u201998), 191\u2013197."},{"key":"e_1_3_1_10_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases","author":"Chen Tsong Yueh","year":"1998","unstructured":"Tsong Yueh Chen, Shing-Chi Cheung, and Siu-Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01. Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong."},{"issue":"1","key":"e_1_3_1_11_2","first-page":"4:1","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen Tsong Yueh","year":"2018","unstructured":"Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys 51, 1 (2018), 4:1\u20134:27.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2016.176"},{"key":"e_1_3_1_13_2","first-page":"1","volume-title":"Proceedings of the 1st Workshop on End-User Software Engineering (WEUSE I \u201905)","author":"Chen Tsong Yueh","year":"2005","unstructured":"Tsong Yueh Chen, Fei-Ching Kuo, and Zhi Quan Zhou. 2005. An effective testing method for end-user programmers. In Proceedings of the 1st Workshop on End-User Software Engineering (WEUSE I \u201905), 1\u20135."},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.69"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2003.1214323"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.07.037"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.23"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473136"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.415"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST57152.2023.00023"},{"key":"e_1_3_1_21_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/MET52542.2021.00012","volume-title":"Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921)","author":"Deng Yao","year":"2021","unstructured":"Yao Deng, Guannan Lou, Xi Zheng, Tianyi Zhang, Miryung Kim, Huai Liu, Chen Wang, and Tsong Yueh Chen. 2021. BMT: Behavior driven development-based metamorphic testing for autonomous driving models. In Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921). IEEE, 32\u201336."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3206427"},{"key":"e_1_3_1_23_2","first-page":"263","volume-title":"Proceedings of the 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC \u201916), Vol. 1","author":"Ding Junhua","year":"2016","unstructured":"Junhua Ding and Dongmei Zhang. 2016. An approach for iteratively generating adequate tests in metamorphic testing: A case study. In Proceedings of the 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC \u201916), Vol. 1, IEEE, 263\u2013268."},{"issue":"4","key":"e_1_3_1_24_2","first-page":"437","article-title":"Case studies on testing with compositional metamorphic relations","volume":"24","author":"Dong Guo-Wei","year":"2008","unstructured":"Guo-Wei Dong, Bao-Wen Xu, Lin Chen, Chang-Hai Nie, and Lu-Lu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University 24, 4 (2008), 437\u2013443.","journal-title":"Journal of Southeast University"},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1007\/978-3-031-21388-5_29","volume-title":"Product-Focused Software Process Improvement","author":"Duque-Torres Alejandra","year":"2022","unstructured":"Alejandra Duque-Torres and Dietmar Pfahl. 2022. Inferring metamorphic relations from JavaDocs: A deep dive Into the MeMo approach. In Product-Focused Software Process Improvement. Davide Taibi, Marco Kuhrmann, Tommi Mikkonen, Jil Kl\u00fcnder, and Pekka Abrahamsson (Eds.), Springer International Publishing, Cham, 418\u2013432."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58846.2023.00081"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00132"},{"key":"e_1_3_1_28_2","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1145\/3213846.3213858","volume-title":"Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201918)","author":"Dwarakanath Anurag","year":"2018","unstructured":"Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder. 2018. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201918). ACM, 118\u2013128."},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/MODELS-C59198.2023.00021"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.59"},{"key":"e_1_3_1_31_2","first-page":"14","volume-title":"Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET \u201918)","author":"Hardin Bonnie","year":"2018","unstructured":"Bonnie Hardin and Upulee Kanewala. 2018. Using semi-supervised learning for predicting metamorphic relations. In Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET \u201918). IEEE, 14\u201317."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1137\/0202009"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1109\/ICRMS55680.2022.9944595","volume-title":"Proceedings of the 13th International Conference on Reliability, Maintainability, and Safety (ICRMS \u201922)","author":"Hong Jie","year":"2022","unstructured":"Jie Hong, Jie Zhang, Qi Qiu, Angang Ma, Meng Li, Shiyu Yan, and Helin Gong. 2022. A dynamic recognition method of metamorphic relation identification. In Proceedings of the 13th International Conference on Reliability, Maintainability, and Safety (ICRMS \u201922), 81\u201386."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSA56465.2022.00030"},{"key":"e_1_3_1_35_2","unstructured":"ISO\/IEC 25010:2011 2011. Systems and software engineering\u2013Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2013System and software quality models. Standard. International Organization for Standardization International Electrotechnical Commission."},{"key":"e_1_3_1_36_2","unstructured":"ISO\/IEC\/IEEE 29119-4:2021(en) 2021. Software and systems engineering\u2013Software testing\u2013Part 4: Test techniques. Standard. International Organization for Standardization International Electrotechnical Commission Institute of Electrical and Electronics Engineers."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/MET.2019.00017"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2014.62"},{"key":"e_1_3_1_39_2","first-page":"1","volume-title":"Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE \u201913)","author":"Kanewala Upulee","year":"2013","unstructured":"Upulee Kanewala and James M. Bieman. 2013. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE \u201913). IEEE, 1\u201310."},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1594"},{"issue":"3","key":"e_1_3_1_41_2","first-page":"21:1","article-title":"The state of the art in end-user software engineering","volume":"43","author":"Ko Amy J.","year":"2011","unstructured":"Amy J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, et al. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3 (2011), 21:1\u201321:44.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_1_42_2","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1109\/MET52542.2021.00017","volume-title":"Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921)","author":"Lascu Andrei","year":"2021","unstructured":"Andrei Lascu, Matt Windsor, Alastair F. Donaldson, Tobias Grosser, and John Wickerson. 2021. Dreaming up metamorphic relations: Experiences from three fuzzer tools. In Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921), 61\u201368."},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2818"},{"key":"e_1_3_1_45_2","first-page":"254","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion \u201921)","author":"Lin Xuanyi","year":"2021","unstructured":"Xuanyi Lin, Zedong Peng, Nan Niu, Wentao Wang, and Hui Liu. 2021. Finding metamorphic relations for scientific software. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion \u201921). IEEE, 254\u2013255."},{"key":"e_1_3_1_46_2","first-page":"1","volume-title":"Proceedings of the 2018 IEEE\/ACM 13th International Workshop on Software Engineering for Science (SE4Science \u201918)","author":"Lin Xuanyi","year":"2018","unstructured":"Xuanyi Lin, Michelle Simon, and Nan Niu. 2018. Hierarchical metamorphic relations for testing scientific software. In Proceedings of the 2018 IEEE\/ACM 13th International Workshop on Software Engineering for Science (SE4Science \u201918). IEEE, 1\u20138."},{"key":"e_1_3_1_47_2","first-page":"26","volume-title":"Proceedings of the 2016 IEEE\/ACM 1st International Workshop on Metamorphic Testing (MET \u201916)","author":"Lindvall Mikael","year":"2016","unstructured":"Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. 2016. Agile metamorphic model-based testing. In Proceedings of the 2016 IEEE\/ACM 1st International Workshop on Metamorphic Testing (MET \u201916). IEEE, 26\u201332."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2012.10"},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1145\/2591062.2591109","volume-title":"Companion Proceedings of the 36th International Conference on Software Engineering (ICSE \u201914)","author":"Liu Huai","year":"2014","unstructured":"Huai Liu, Iman I. Yusuf, Heinz W. Schmidt, and Tsong Yueh Chen. 2014. Metamorphic fault tolerance: An automated and systematic methodology for fault tolerance in the absence of test oracle. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE \u201914), 420\u2013423."},{"issue":"2","key":"e_1_3_1_51_2","first-page":"50:1","article-title":"Generation-based differential fuzzing for deep learning libraries","volume":"33","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Yuheng Huang, Zhijie Wang, Lei Ma, Chunrong Fang, Mingzheng Gu, Xufan Zhang, and Zhenyu Chen. 2023. Generation-based differential fuzzing for deep learning libraries. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 50:1\u201350:28.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_52_2","first-page":"23","volume-title":"Proceedings of the 2022 IEEE\/ACM 7th International Workshop on Metamorphic Testing (MET \u201922)","author":"Luu Quang-Hung","year":"2022","unstructured":"Quang-Hung Luu, Huai Liu, Tsong Yueh Chen, and Hai L. Vu. 2022. Testing ocean software with metamorphic testing. In Proceedings of the 2022 IEEE\/ACM 7th International Workshop on Metamorphic Testing (MET \u201922), 23\u201330."},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2024.3370740"},{"issue":"2","key":"e_1_3_1_54_2","first-page":"37e:1","article-title":"Software engineering for AI-based systems: A survey","volume":"31","author":"Mart\u00ednez-Fern\u00e1ndez Silverio","year":"2022","unstructured":"Silverio Mart\u00ednez-Fern\u00e1ndez, Justus Bogner, Xavier Franch, Marc Oriol, Julien Siebert, Adam Trendowicz, Anna Maria Vollmer, and Stefan Wagner. 2022. Software engineering for AI-based systems: A survey. ACM Transactions on Software Engineering and Methodology 31, 2 (2022), 37e:1\u201337e:59.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.57623"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340482.3342741"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/s41870-023-01390-9"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643747"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/62959.62964"},{"key":"e_1_3_1_60_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/B978-0-12-800160-8.00001-2","article-title":"Automated test oracles: A survey","volume":"95","author":"Pezz\u00e8 Mauro","year":"2014","unstructured":"Mauro Pezz\u00e8 and Cheng Zhang. 2014. Automated test oracles: A survey. In Advances in Computers. Atif Memon (Ed.), Vol. 95, Academic Press, Waltham, MA, 1\u201348.","journal-title":"Advances in Computers"},{"issue":"4","key":"e_1_3_1_61_2","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1145\/1721654.1721687","article-title":"CHOC\u2019LATE: A framework for specification-based testing","volume":"53","author":"Poon Pak-Lok","year":"2010","unstructured":"Pak-Lok Poon, Sau-Fun Tang, T. H. Tse, and Tsong Yueh Chen. 2010. CHOC\u2019LATE: A framework for specification-based testing. Communications of the ACM 53, 4 (2010), 113\u2013118.","journal-title":"Communications of the ACM"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3009698"},{"key":"e_1_3_1_63_2","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1145\/3387940.3392250","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops","author":"Rahman Karishma","year":"2020","unstructured":"Karishma Rahman, Indika Kahanda, and Upulee Kanewala. 2020. MRpredT: Using text mining for metamorphic relation prediction. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops, 420\u2013424."},{"key":"e_1_3_1_64_2","first-page":"10","volume-title":"Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET \u201918)","author":"Rahman Karishma","year":"2018","unstructured":"Karishma Rahman and Upulee Kanewala. 2018. Predicting metamorphic relations for matrix calculation programs. In Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET \u201918). IEEE, 10\u201313."},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/MET52542.2021.00015","volume-title":"Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921)","author":"Raunak M. S.","year":"2021","unstructured":"M. S. Raunak and Megan M. Olsen. 2021. Metamorphic testing on the continuum of verification and validation of simulation models. In Proceedings of the 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921). IEEE, 47\u201352."},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1566"},{"key":"e_1_3_1_67_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/MET.2017.3","volume-title":"Proceedings of the 2017 IEEE\/ACM 2nd International Workshop on Metamorphic Testing (MET \u201917)","author":"Segura Sergio","year":"2017","unstructured":"Sergio Segura, Amador Dur\u00e1n, Javier Troya, and Antonio Ruiz Cort\u00e9s. 2017. A template-based approach to describing metamorphic relations. In Proceedings of the 2017 IEEE\/ACM 2nd International Workshop on Metamorphic Testing (MET \u201917). IEEE, 3\u20139."},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/1138929.1138938"},{"key":"e_1_3_1_71_2","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1145\/1993498.1993539","volume-title":"Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI \u201911)","author":"Srivastava Varun","year":"2011","unstructured":"Varun Srivastava, Michael D. Bond, Kathryn S. McKinley, and Vitaly Shmatikov. 2011. A security policy oracle: Detecting security holes using multiple API implementations. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI \u201911), 343\u2013354."},{"issue":"1","key":"e_1_3_1_72_2","first-page":"8:1","article-title":"An interleaving guided metamorphic testing approach for concurrent programs","volume":"33","author":"Sun Chang-Ai","year":"2023","unstructured":"Chang-Ai Sun, Hepeng Dai, Ning Geng, Huai Liu, Tsong Yueh Chen, Peng Wu, Yan Cai, and Jinqiu Wang. 2023. An interleaving guided metamorphic testing approach for concurrent programs. ACM Transactions on Software Engineering and Methodology 33, 1 (2023), 8:1\u20138:21.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"issue":"1","key":"e_1_3_1_73_2","first-page":"20:1","article-title":"Feedback-directed metamorphic testing","volume":"31","author":"Sun Chang-Ai","year":"2023","unstructured":"Chang-Ai Sun, Hepeng Dai, Huai Liu, and Tsong Yueh Chen. 2023. Feedback-directed metamorphic testing. ACM Transactions on Software Engineering and Methodology 31, 1 (2023), 20:1\u201320:34.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"issue":"9","key":"e_1_3_1_74_2","first-page":"1764","article-title":"METRIC  \\({}^{+}\\) : A metamorphic relation identification technique based on input plus output domains","volume":"47","author":"Sun Chang-Ai","year":"2019","unstructured":"Chang-Ai Sun, An Fu, Pak-Lok Poon, Xiaoyuan Xie, Huai Liu, and Tsong Yueh Chen. 2019. METRIC \\({}^{+}\\) : A metamorphic relation identification technique based on input plus output domains. IEEE Transactions on Software Engineering 47, 9 (2019), 1764\u20131785.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.3280"},{"key":"e_1_3_1_76_2","first-page":"12","volume-title":"Proceedings of the 2016 IEEE\/ACM 1st International Workshop on Metamorphic Testing (MET \u201916)","author":"Sun Chang-Ai","year":"2016","unstructured":"Chang-Ai Sun, Yiqiang Liu, Zuoyi Wang, and Wing Kwong Chan. 2016. \\(\\mu\\) MT: A data mutation directed metamorphic relation acquisition methodology. In Proceedings of the 2016 IEEE\/ACM 1st International Workshop on Metamorphic Testing (MET \u201916). IEEE, 12\u201318."},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3462722"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409758"},{"key":"e_1_3_1_79_2","first-page":"85","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion \u201921)","author":"Terragni Valerio","year":"2021","unstructured":"Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezze. 2021. GAssert: A fully automated Tool to improve assertion oracles. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion \u201921). IEEE, 85\u201388."},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.05.043"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.3390\/info10120392"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.08.008"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3656340"},{"key":"e_1_3_1_86_2","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1145\/3387940.3392251","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops","author":"Yang Yingzhuo","year":"2020","unstructured":"Yingzhuo Yang and Chang Xu. 2020. M.R. Hunter: Hunting for metamorphic relations by puzzle solving. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops, 404\u2013409."},{"issue":"6","key":"e_1_3_1_87_2","first-page":"e12902:1","article-title":"Syntax-based metamorphic relation prediction via the bagging framework","volume":"39","author":"Yin Yuyu","year":"2022","unstructured":"Yuyu Yin, Jiajie Ruan, Youhuizi Li, Yu Li, and Zhijin Pan. 2022. Syntax-based metamorphic relation prediction via the bagging framework. Expert Systems 39, 6 (2022), e12902:1\u2013e12902:21.","journal-title":"Expert Systems"},{"key":"e_1_3_1_88_2","first-page":"1","article-title":"A multi-label metamorphic relations prediction approach based on RBF neural network","volume":"12","author":"Zeng Jin-wei","year":"2016","unstructured":"Jin-wei Zeng, Pengcheng Zhang, Wenrui Li, and Xinli Zhou. 2016. A multi-label metamorphic relations prediction approach based on RBF neural network. Computer and Modernization 12 (2016), 1\u20136.","journal-title":"Computer and Modernization"},{"key":"e_1_3_1_89_2","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1109\/ICSME.2019.00035","volume-title":"Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201919)","author":"Zhang Bo","year":"2019","unstructured":"Bo Zhang, Hongyu Zhang, Junjie Chen, Dan Hao, and Pablo Moscato. 2019. Automatic discovery and cleansing of numerical metamorphic relations. In Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201919). IEEE, 235\u2013245."},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_1_91_2","first-page":"1","volume-title":"Proceedings of the 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201925)","author":"Zhang Jiaming","year":"2025","unstructured":"Jiaming Zhang, Chang-Ai Sun, Huai Liu, and Sijin Dong. 2025. Can large language models discover metamorphic relations? A large-scale empirical study. In Proceedings of the 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201925), 1\u201312."},{"key":"e_1_3_1_92_2","first-page":"399","volume-title":"Proceedings of the 2016 17th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD \u201916)","author":"Zhang Xinglong","year":"2016","unstructured":"Xinglong Zhang, Lei Yu, and Xuemei Hou. 2016. A method of metamorphic relations constructing for object-oriented software testing. In Proceedings of the 2016 17th IEEE\/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD \u201916). IEEE, 399\u2013406."},{"key":"e_1_3_1_93_2","first-page":"1780","volume-title":"Proceedings of the 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC \u201923)","author":"Zhang Yifan","year":"2023","unstructured":"Yifan Zhang, Dave Towey, and Matthew Pike. 2023. Automated metamorphic-relation generation with ChatGPT: An experience report. In Proceedings of the 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC \u201923), 1780\u20131785."},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695990"},{"key":"e_1_3_1_95_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1109\/MET52542.2021.00018","volume-title":"Proceedings of 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921)","author":"Zhou Zenghui","year":"2021","unstructured":"Zenghui Zhou, Zheng Zheng, Tsong Yueh Chen, Jinyi Zhou, and Kun Qiu. 2021. Follow-up test cases are better than source test cases in metamorphic testing: A preliminary study. In Proceedings of 2021 IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921), 69\u201374."},{"key":"e_1_3_1_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876433"},{"key":"e_1_3_1_97_2","volume-title":"Automated Functional Testing of Web Search Engines in the Absence of an Oracle","author":"Zhou Zhi Quan","year":"2007","unstructured":"Zhi Quan Zhou, T. H. Tse, Fei-Ching Kuo, and Tsong Yueh Chen. 2007. Automated Functional Testing of Web Search Engines in the Absence of an Oracle. Technical Report TR-2007-06. Department of Computer Science, The University of Hong Kong."},{"key":"e_1_3_1_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2478001"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.437"},{"key":"e_1_3_1_100_2","doi-asserted-by":"publisher","DOI":"10.14778\/3636218.3636236"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708521","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708521","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:45Z","timestamp":1750295865000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708521"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,27]]},"references-count":99,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3708521"],"URL":"https:\/\/doi.org\/10.1145\/3708521","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,27]]},"assertion":[{"value":"2024-03-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}