{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T19:42:22Z","timestamp":1774640542635,"version":"3.50.1"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T00:00:00Z","timestamp":1675123200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61872039"],"award-info":[{"award-number":["61872039"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Australian Research Council Discovery Project","award":["DP210102447"],"award-info":[{"award-number":["DP210102447"]}]},{"name":"Beijing Natural Science Foundation","award":["4162040"],"award-info":[{"award-number":["4162040"]}]},{"name":"Aeronautical Science Foundation of China","award":["2016ZD74004"],"award-info":[{"award-number":["2016ZD74004"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","award":["FRF-GF-19-B19"],"award-info":[{"award-number":["FRF-GF-19-B19"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,1,31]]},"abstract":"<jats:p>Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a set of metamorphic relations among multiple software inputs and their expected outputs, metamorphic testing not only provides a test case generation strategy by constructing new (or follow-up) test cases from some original (or source) test cases, but also a test result verification mechanism through checking the relationship between the outputs of source and follow-up test cases. Many efforts have been made to further improve the cost-effectiveness of metamorphic testing from different perspectives. Some studies attempted to identify \u201cgood\u201d metamorphic relations, while other studies were focused on applying effective test case generation strategies especially for source test cases. In this article, we propose improving the cost-effectiveness of metamorphic testing by leveraging the feedback information obtained in the test execution process. Consequently, we develop a new approach, namely feedback-directed metamorphic testing, which makes use of test execution information to dynamically adjust the selection of metamorphic relations and selection of source test cases. We conduct an empirical study to evaluate the proposed approach based on four laboratory programs, one GNU program, and one industry program. The empirical results show that feedback-directed metamorphic testing can use fewer test cases and take less time than the traditional metamorphic testing for detecting the same number of faults. It is clearly demonstrated that the use of feedback information about test execution does help enhance the cost-effectiveness of metamorphic testing. Our work provides a new perspective to improve the efficacy and applicability of metamorphic testing as well as many other software testing techniques.<\/jats:p>","DOI":"10.1145\/3533314","type":"journal-article","created":{"date-parts":[[2022,5,24]],"date-time":"2022-05-24T07:10:54Z","timestamp":1653376254000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Feedback-Directed Metamorphic Testing"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3696-6176","authenticated-orcid":false,"given":"Chang-Ai","family":"Sun","sequence":"first","affiliation":[{"name":"University of Science and Technology Beijing, Haidian District, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5507-3053","authenticated-orcid":false,"given":"Hepeng","family":"Dai","sequence":"additional","affiliation":[{"name":"University of Science and Technology Beijing, Haidian District, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3125-4399","authenticated-orcid":false,"given":"Huai","family":"Liu","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, John Street, Hawthorn, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3578-0994","authenticated-orcid":false,"given":"Tsong Yueh","family":"Chen","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, John Street, Hawthorn, Australia"}]}],"member":"320","published-online":{"date-parts":[[2023,2,13]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"140","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP\u201921)","author":"Ahlgren John","year":"2021","unstructured":"John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Erik Meijer, Silvia Sapora, and Justin Spahr. 2021. Testing web enabled simulation at scale using metamorphic testing. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP\u201921). 140\u2013149."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_4_2","first-page":"147","volume-title":"Proceedings of the 15th International Conference on Secure Software Integration and Reliability Improvement (SSIRI\u201911)","author":"Asrafi Mahmuda","year":"2011","unstructured":"Mahmuda Asrafi, Huai Liu, and Fei-Ching Kuo. 2011. On testing effectiveness of metamorphic relations: A case study. In Proceedings of the 15th International Conference on Secure Software Integration and Reliability Improvement (SSIRI\u201911). 147\u2013156."},{"key":"e_1_3_2_5_2","first-page":"1264","volume-title":"Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921)","author":"Ayerdi Jon","year":"2021","unstructured":"Jon Ayerdi, Valerio Terragni, Aitor Arrieta, Paolo Tonella, Goiuria Sagardui, and Maite Arratibel. 2021. Generating metamorphic relations for cyber-physical systems with genetic programming: An industrial case study. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921). 1264\u20131274."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2016.2547380"},{"key":"e_1_3_2_8_2","first-page":"5","volume-title":"Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916), Co-Located with the 38th International Conference on Software Engineering (ICSE\u201916)","author":"Barus Arlinta Christy","year":"2016","unstructured":"Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Heinz W. Schmidt. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET\u201916), Co-Located with the 38th International Conference on Software Engineering (ICSE\u201916). 5\u201311."},{"key":"e_1_3_2_9_2","volume-title":"SWEBOK: Guide to the Software Engineering Body of Knowledge (version 3.0 ed.)","author":"Bourque Pierre","year":"2014","unstructured":"Pierre Bourque and Richard E. Fairley (Eds.). 2014. SWEBOK: Guide to the Software Engineering Body of Knowledge (version 3.0 ed.). IEEE Computer Society, Los Alamitos, CA."},{"key":"e_1_3_2_10_2","first-page":"408","volume-title":"Proceedings of the 20th International Symposium on Fault-Tolerant Computing (FTCS\u201990)","author":"Brilliant Susan S.","year":"1990","unstructured":"Susan S. Brilliant, John C. Knight, and P. E. Ammann. 1990. On the performance of software testing using multiple versions. In Proceedings of the 20th International Symposium on Fault-Tolerant Computing (FTCS\u201990). 408\u2013415."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(02)00108-8"},{"key":"e_1_3_2_12_2","first-page":"240","volume-title":"Proceedings of the 7th IEEE International Symposium on High Assurance Systems Engineering (HSE\u201902)","author":"Cai Kai-Yuan","year":"2002","unstructured":"Kai-Yuan Cai, Tsong Yueh Chen, and T. H. Tse. 2002. Towards research on software cybernetics. In Proceedings of the 7th IEEE International Symposium on High Assurance Systems Engineering (HSE\u201902). 240\u2013241."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.11.008"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470050118.ecse707"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3072166"},{"key":"e_1_3_2_16_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases","author":"Chen Tsong Yueh","year":"1998","unstructured":"Tsong Yueh Chen, Shing C. Cheung, and Shiu Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report. Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, Tech. Rep. HKUST-CS98-01."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-10-24"},{"key":"e_1_3_2_18_2","first-page":"569","volume-title":"Proceedings of the 4th lberoAmerican Symposium on Software Engineering and Knowledge Engineer-ing (JIISIC\u201904)","author":"Chen Tsong Yueh","year":"2004","unstructured":"Tsong Yueh Chen, D. H. Huang, T. H. Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th lberoAmerican Symposium on Software Engineering and Knowledge Engineer-ing (JIISIC\u201904). 569\u2013583."},{"issue":"1","key":"e_1_3_2_19_2","first-page":"4:1\u20134:27","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen Tsong Yueh","year":"2018","unstructured":"Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys 51, 1 (2018), 4:1\u20134:27.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_20_2","first-page":"128","volume-title":"Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD\u201904)","author":"Chen Tsong Yueh","year":"2004","unstructured":"Tsong Yueh Chen, Fei-Ching Kuo, Ying Liu, and Antony Tang. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD\u201904). 128\u2013134."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2016.176"},{"key":"e_1_3_2_22_2","first-page":"320","volume-title":"Proceedings of the 9th Asian Computing Science Conference (ASIAN\u201904)","author":"Chen Tsong Yueh","year":"2004","unstructured":"Tsong Yueh Chen, Hing Leung, and Ieng Kei Mak. 2004. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference (ASIAN\u201904). 320\u2013329."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.07.037"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473136"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.368132"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEW.2005.33"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-005-3861-2"},{"key":"e_1_3_2_28_2","first-page":"193","volume-title":"Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS\u201913)","author":"Dong Guowei","year":"2013","unstructured":"Guowei Dong, Tao Guo, and Puhan Zhang. 2013. Security assurance with program path analysis and metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS\u201913). 193\u2013197."},{"key":"e_1_3_2_29_2","unstructured":"GNU Project. 2006. GNU grep. (2006). Retrieved from http:\/\/www.gnu.org\/software\/grep."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337242"},{"key":"e_1_3_2_31_2","volume-title":"Proceedings of the Encyclopedia of Software Engineering","author":"Hamlet Richard","year":"2002","unstructured":"Richard Hamlet. 2002. Random testing. In Proceedings of the Encyclopedia of Software Engineering. John Wiley & Sons, Inc."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397382"},{"key":"e_1_3_2_34_2","volume-title":"Statistical Methods for Meta-Analysis","author":"Hedges Larry","year":"2014","unstructured":"Larry Hedges and Ingram Olkin. 2014. Statistical Methods for Meta-Analysis. Academic Press."},{"key":"e_1_3_2_35_2","first-page":"58","volume-title":"Proceedings of the 20th International Symposium on Trends in Functional Programming (TFP\u201919) (Lecture Notes in Computer Science)","volume":"12053","author":"Hughes John","year":"2019","unstructured":"John Hughes. 2019. How to specify it!. In Proceedings of the 20th International Symposium on Trends in Functional Programming (TFP\u201919) (Lecture Notes in Computer Science), Vol. 12053. 58\u201383."},{"key":"e_1_3_2_36_2","first-page":"90","volume-title":"Proceedings of the 28th International Symposium on Software Reliability Engineering Workshops (ISSREW\u201917)","author":"Hui Zhan-Wei","year":"2017","unstructured":"Zhan-Wei Hui, Song Huang, Tsong Yueh Chen, Man F. Lau, and Sebastian Ng. 2017. Identifying failed test cases through metamorphic testing. In Proceedings of the 28th International Symposium on Software Reliability Engineering Workshops (ISSREW\u201917). 90\u201391."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1808266.1808280"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213874"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238176"},{"key":"e_1_3_2_40_2","first-page":"466","volume-title":"Proceedings of the 39th International Computer Software and Applications Conference (COMPSAC\u201915)","volume":"3","author":"Li Ye","year":"2015","unstructured":"Ye Li, Bei-Bei Yin, Junpeng Lv, and Kai-Yuan Cai. 2015. Approach for test profile optimization in dynamic random testing. In Proceedings of the 39th International Computer Software and Applications Conference (COMPSAC\u201915), Vol. 3. 466\u2013471."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_3_2_42_2","first-page":"59","volume-title":"Proceedings of the 12th International Conference on Quality Software (QSIC\u201912)","author":"Liu Huai","year":"2012","unstructured":"Huai Liu, Xuan Liu, and Tsong Yueh Chen. 2012. A new method for constructing metamorphic relations. In Proceedings of the 12th International Conference on Quality Software (QSIC\u201912). 59\u201368."},{"key":"e_1_3_2_43_2","first-page":"19","volume-title":"Proceedings of the 35th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW\u201911)","author":"Lv Junpeng","year":"2011","unstructured":"Junpeng Lv, Hai Hu, and Kai-Yuan Cai. 2011. A sufficient condition for parameters estimation in dynamic random testing. In Proceedings of the 35th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW\u201911). 19\u201324."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2310194"},{"key":"e_1_3_2_45_2","first-page":"458","volume-title":"Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI\u201920)","author":"Ma Pingchuan","year":"2020","unstructured":"Pingchuan Ma, Shuai Wang, and Jin Liu. 2020. Metamorphic testing and certified mitigation of fairness violations in NLP models. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI\u201920). 458\u2013465."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.308"},{"key":"e_1_3_2_47_2","first-page":"186","volume-title":"Proceedings of the 13th IEEE International Conference on Software Testing, Validation and Verification (ICST\u201920)","author":"Mai Phu X.","year":"2020","unstructured":"Phu X. Mai, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2020. Metamorphic security testing for web systems. In Proceedings of the 13th IEEE International Conference on Software Testing, Validation and Verification (ICST\u201920). 186\u2013197."},{"key":"e_1_3_2_48_2","first-page":"639","volume-title":"Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921)","author":"Mansur Muhammad Numair","year":"2021","unstructured":"Muhammad Numair Mansur, Maria Christakis, and Valentin W\u00fcstholz. 2021. Metamorphic testing of datalog engines. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921). 639\u2013650."},{"key":"e_1_3_2_49_2","first-page":"101","volume-title":"Proceedings of the 1st IEEE International Conference On Artificial Intelligence Testing (AITest\u201919)","author":"Marijan Dusica","year":"2019","unstructured":"Dusica Marijan, Arnaud Gotlieb, and Mohit Kumar Ahuja. 2019. Challenges of testing machine learning based systems. In Proceedings of the 1st IEEE International Conference On Artificial Intelligence Testing (AITest\u201919). 101\u2013102."},{"key":"e_1_3_2_50_2","first-page":"475","volume-title":"Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC\u201906)","author":"Mayer Johannes","year":"2006","unstructured":"Johannes Mayer and Ralph Guderlei. 2006. An empirical study on the selection of good metamorphic relations. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC\u201906). 475\u2013484."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376420"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/62959.62964"},{"key":"e_1_3_2_53_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1145\/1390630.1390643","volume-title":"Proceedings of the 7th International Symposium on Software Testing and Analysis (ISSTA\u201908)","author":"Pacheco Carlos","year":"2008","unstructured":"Carlos Pacheco, Shuvendu K Lahiri, and Thomas Ball. 2008. Finding errors in. net with feedback-directed random testing. In Proceedings of the 7th International Symposium on Software Testing and Analysis (ISSTA\u201908). 87\u201396."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.37"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-017-9392-4"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3009698"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"issue":"2","key":"e_1_3_2_58_2","first-page":"467","article-title":"New effect size rules of thumb","volume":"8","author":"Sawilowsky Shlomo S.","year":"2009","unstructured":"Shlomo S. Sawilowsky. 2009. New effect size rules of thumb. Journal of Modern Applied Statistical Methods 8, 2 (2009), 467\u2013474.","journal-title":"Journal of Modern Applied Statistical Methods"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2757667.2757677"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_3_2_62_2","first-page":"110574:1\u2013110574","article-title":"Adaptive metamorphic testing with contextual bandits","volume":"165","author":"Spieker Helge","year":"2020","unstructured":"Helge Spieker and Arnaud Gotlieb. 2020. Adaptive metamorphic testing with contextual bandits. Journal of Systems and Software 165 (2020), 110574:1\u2013110574:14.","journal-title":"Journal of Systems and Software"},{"issue":"2","key":"e_1_3_2_63_2","first-page":"157","article-title":"Adaptive partition testing","volume":"68","author":"Sun Chang-ai","year":"2018","unstructured":"Chang-ai Sun, Hepeng Dai, Huai Liu, Tsong Yueh Chen, and Kai-Yuan Cai. 2018. Adaptive partition testing. IEEE Transactions on Computers 68, 2 (2018), 157\u2013169.","journal-title":"IEEE Transactions on Computers"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2019.2960496"},{"issue":"9","key":"e_1_3_2_65_2","first-page":"1765","article-title":"METRIC+: A metamorphic relation identification technique based on input plus output domains","volume":"47","author":"Sun Chang-ai","year":"2021","unstructured":"Chang-ai Sun, An Fu, Pak-Lok Poon, Xiaoyuan Xie, Huai Liu, and Tsong Yueh Chen. 2021. METRIC+: A metamorphic relation identification technique based on input plus output domains. IEEE Transactions on Software Engineering 47, 9 (2021), 1765\u20131786.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_66_2","first-page":"12","volume-title":"Proceedings of 2016 IEEE\/ACM the 1st International Workshop on Metamorphic Testing (MET\u201916), Co-Located with the 38th International Conference on Software Engineering (ICSE\u201916)","author":"Sun Chang-ai","year":"2016","unstructured":"Chang-ai Sun, Yiqiang Liu, Zuoyi Wang, and W. K. Chan. 2016. \\(\\mu\\) MT: A data mutation directed metamorphic relation acquisition methodology. In Proceedings of 2016 IEEE\/ACM the 1st International Workshop on Metamorphic Testing (MET\u201916), Co-Located with the 38th International Conference on Software Engineering (ICSE\u201916). 12\u201318."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-014-3040-y"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.08.008"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.08.095"},{"key":"e_1_3_2_71_2","first-page":"37","volume-title":"Proceedings of the 38th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW\u201914)","author":"Yang Zijiang","year":"2014","unstructured":"Zijiang Yang, Beibei Yin, Junpeng Lv, Kai-Yuan Cai, Stephen S. Yau, and Jia Yu. 2014. Dynamic random testing with parameter adjustment. In Proceedings of the 38th IEEE Annual International Computer Software and Applications Conference Workshops (COMPSACW\u201914). 37\u201342."},{"key":"e_1_3_2_72_2","first-page":"16908","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201921)","author":"Yuan Yuanyuan","year":"2021","unstructured":"Yuanyuan Yuan, Shuai Wang, Mingyue Jiang, and Tsong Yueh Chen. 2021. Perception matters: Detecting perception failures of VQA models using metamorphic testing. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201921). 16908\u201316917."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_75_2","first-page":"31","volume-title":"Proceedings of the 38th International Computer Software and Applications Conference Workshops (COMPSACW\u201914)","author":"Zhang Lei","year":"2014","unstructured":"Lei Zhang, Bei-Bei Yin, Junpeng Lv, Kai-Yuan Cai, Stephen S. Yau, and Jia Yu. 2014. A history-based dynamic random software testing. In Proceedings of the 38th International Computer Software and Applications Conference Workshops (COMPSACW\u201914). 31\u201336."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533314","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533314","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:17Z","timestamp":1750186817000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533314"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,31]]},"references-count":74,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,31]]}},"alternative-id":["10.1145\/3533314"],"URL":"https:\/\/doi.org\/10.1145\/3533314","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,31]]},"assertion":[{"value":"2021-08-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}