{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T05:05:21Z","timestamp":1769749521969,"version":"3.49.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,1,18]],"date-time":"2025-01-18T00:00:00Z","timestamp":1737158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62372021"],"award-info":[{"award-number":["62372021"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP210102447"],"award-info":[{"award-number":["DP210102447"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,2,28]]},"abstract":"<jats:p>Metamorphic testing, thanks to its high failure-detection effectiveness especially in the absence of test oracle, has been widely applied in both the traditional context of software testing and other relevant fields such as fault localization and program repair. Its core element is a set of metamorphic relations, which are the necessary properties of the target algorithm in the form of the relationships among multiple inputs and corresponding expected outputs. When a relation is violated by the outputs of a group of test cases, namely metamorphic group of test cases, that are constructed based on the relation, a failure is said to be revealed. Traditionally, the primary task of software testing is to reveal failures. Therefore, from the perspective of software testing, it may not need to know which test case(s) in the metamorphic group cause the violation and thus the failure. However, such information is definitely helpful for other software engineering activities, such as software debugging. The current literature of metamorphic testing lacks a systematic mechanism of identifying the actual failure-revealing test cases, which hinders its applicability and effectiveness in other relevant fields. In this article, we propose a new technique for the FAILure-revealing Test case Identification in Metamorphic testing, namely FAILTIM. The approach is based on a novel application of statistical methods. More specifically, we leverage and adapt the basic ideas of spectrum-based techniques, which are originally used in fault localization, and propose the utilization of a set of risk formulas to estimate the suspiciousness of each individual test case in metamorphic groups. Failure-revealing test cases are then suggested according to their suspiciousness. A series of experiments have been conducted to evaluate the effectiveness and efficiency of FAILTIM using 9 subject programs and 30 risk formulas. The experimental results showed that the new approach can achieve a high accuracy in identifying the actual failure-revealing test cases in metamorphic testing. Consequently, our study will help boost the applicability and performance of metamorphic testing beyond testing to other software engineering areas. The present work also unfolds a number of research directions for further advancing the theory of metamorphic testing and more broadly, software testing.<\/jats:p>","DOI":"10.1145\/3695990","type":"journal-article","created":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T13:56:18Z","timestamp":1726235778000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Identifying the Failure-Revealing Test Cases in Metamorphic Testing: A Statistical Approach"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7922-9067","authenticated-orcid":false,"given":"Zheng","family":"Zheng","sequence":"first","affiliation":[{"name":"Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4291-2601","authenticated-orcid":false,"given":"Daixu","family":"Ren","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3125-4399","authenticated-orcid":false,"given":"Huai","family":"Liu","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3578-0994","authenticated-orcid":false,"given":"Tsong Yueh","family":"Chen","sequence":"additional","affiliation":[{"name":"Swinburne University of Technology, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9695-9220","authenticated-orcid":false,"given":"Tiancheng","family":"Li","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2025,1,18]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.341844"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1985.231893"},{"key":"e_1_3_1_4_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases","author":"Chen T. Y.","year":"1998","unstructured":"T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS 98\u201301, The Hong Kong University of Science and Technology, Hong Kong."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"issue":"1","key":"e_1_3_1_6_2","first-page":"4:1","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen T. Y.","year":"2018","unstructured":"T. Y. Chen, F. C. Kuo, H. Liu, P. L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Comput. Surv. 51, 1 (2018), 4:1\u20134:27.","journal-title":"ACM Comput. Surv"},{"issue":"1","key":"e_1_3_1_7_2","first-page":"1","article-title":"An innovative approach for testing bioinformatics programs using metamorphic testing","volume":"10","author":"Chen T. Y.","year":"2009","unstructured":"T. Y. Chen, J. W. K. Ho, H. Liu, and X. Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinf. 10, 1 (2009), 1\u201312.","journal-title":"BMC Bioinf."},{"issue":"6","key":"e_1_3_1_8_2","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1142\/S021972001100563X","article-title":"Verification of phylogenetic inference programs using metamorphic testing","volume":"9","author":"Sadi M. S.","year":"2011","unstructured":"M. S. Sadi, F. C. Kuo, J. W. K. Ho, M. A. Charleston, and T. Y. Chen. 2011. Verification of phylogenetic inference programs using metamorphic testing. J. Bioinf. Comput. Biol. 9, 6 (2011), 729\u2013747.","journal-title":"J. Bioinf. Comput. Biol."},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3256322"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00023"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111041"},{"issue":"1","key":"e_1_3_1_13_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3607182","article-title":"An interleaving guided metamorphic testing approach for concurrent programs","volume":"33","author":"Sun C. A.","year":"2023","unstructured":"C. A. Sun, H. Dai, N. Geng, H. Liu, T. Y. Chen, P. Wu, Y. Cai, and J. Wang. 2023. An interleaving guided metamorphic testing approach for concurrent programs. ACM Trans. Softw. Eng. Methodology 33, 1 (2023), 1\u201321.","journal-title":"ACM Trans. Softw. Eng. Methodology"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_3_1_15_2","first-page":"270","volume-title":"Proc. 17th Asia Pacific Softw. Eng. Conf.","author":"Tao Q.","year":"2010","unstructured":"Q. Tao, W. Wu, C. Zhao, and W. Shen. 2010. An automatic testing approach for compiler based on metamorphic testing technique. In Proc. 17th Asia Pacific Softw. Eng. Conf., 270\u2013279."},{"key":"e_1_3_1_16_2","first-page":"1","volume-title":"Proc. ACM Program. Lang","volume":"1","author":"Donaldson A. F.","year":"2017","unstructured":"A. F. Donaldson, H. Evrard, A. Lascu, and P. Thomson. 2017. Automated testing of graphics shader compilers. Proc. ACM Program. Lang. 1, OOPSLA (2017), 1\u201329."},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00201"},{"key":"e_1_3_1_18_2","volume-title":"Properties of Machine Learning Applications for Use in Metamorphic Testing","author":"Murphy C.","year":"2011","unstructured":"C. Murphy, G. E. Kaiser, and L. Hu. 2011. Properties of Machine Learning Applications for Use in Metamorphic Testing. Technical Report CUCS-011-08, Columbia University, U.S."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561157"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2020.2972266"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2478001"},{"key":"e_1_3_1_23_2","first-page":"85","volume-title":"Proc. 6th Int. Conf. Quality Softw.","author":"Mayer J.","year":"2006","unstructured":"J. Mayer and R. Guderlei. 2006. On random testing of image processing applications. In Proc. 6th Int. Conf. Quality Softw., 85\u201392."},{"key":"e_1_3_1_24_2","first-page":"236","article-title":"Dependency-aware metamorphic testing of datalog engines","author":"Mansur M. N.","year":"2023","unstructured":"M. N. Mansur, V. W\u00fcstholz, M. Christakis. 2023. Dependency-aware metamorphic testing of datalog engines. In Proc. 32nd Int. Symp. Softw. Test. Anal., 236\u2013247.","journal-title":"Proc. 32nd Int. Symp. Softw. Test. Anal."},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00202"},{"issue":"2","key":"e_1_3_1_26_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3624742","article-title":"Stress testing control loops in cyber-physical systems","volume":"33","author":"Mandrioli C.","year":"2023","unstructured":"C. Mandrioli, S. Y. Shin, M. Maggio, D. Bianculli, and L Briand. 2023. Stress testing control loops in cyber-physical systems. ACM Trans. Softw. Eng. Methodology 33, 2 (2023), 1\u201358.","journal-title":"ACM Trans. Softw. Eng. Methodology"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241979"},{"issue":"4","key":"e_1_3_1_29_2","first-page":"1965","article-title":"A declarative metamorphic testing framework for autonomous driving","volume":"49","author":"Deng Y.","year":"2022","unstructured":"Y. Deng, X. Zheng, T. Zhang, H. Liu, G. Lou, M. Kim, and T. Y. Chen. 2022. A declarative metamorphic testing framework for autonomous driving. IEEE Trans. Softw. Eng. 49, 4 (2022), 1965\u20131982.","journal-title":"IEEE Trans. Softw. Eng"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.08.008"},{"key":"e_1_3_1_31_2","first-page":"420","volume-title":"Companion Proc. 36th Int. Conf. Softw. Eng.","author":"Liu H.","year":"2014","unstructured":"H. Liu, I. I. Yusuf, H. W. Schmidt, and T. Y. Chen. 2014. Metamorphic fault tolerance: An automated and systematic methodology for fault tolerance in the absence of test oracle. In Companion Proc. 36th Int. Conf. Softw. Eng., 420\u2013423."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010248"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581397"},{"key":"e_1_3_1_34_2","first-page":"595","volume-title":"Proc. 32th IEEE\/IFIP Int. Conf. Depen. Syst. Netw.","author":"Chen M.","year":"2002","unstructured":"M. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer. 2002. Pinpoint: Problem determination in large, dynamic internet services. In Proc. 32th IEEE\/IFIP Int. Conf. Depen. Syst. Netw., 595\u2013604."},{"key":"e_1_3_1_35_2","first-page":"39","volume-title":"Proc. 12th Pacific Rim Int. Symp. Depen. Comput.","author":"Abreu R.","year":"2006","unstructured":"R. Abreu, P. Zoeteweij, and A. Gemund. 2006. An evaluation of similarity coefficients for software fault localization. In Proc. 12th Pacific Rim Int. Symp. Depen. Comput., 39\u201346."},{"issue":"3","key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2000791.2000795","article-title":"A model for spectra-based software diagnosis","volume":"20","author":"Naish L.","year":"2006","unstructured":"L. Naish, H. J. Lee, and K. Ramamohanarao. 2006. A model for spectra-based software diagnosis. ACM Trans. Softw. Eng. Methodology 20, 3 (2011), 1\u201332.","journal-title":"ACM Trans. Softw. Eng. Methodology"},{"key":"e_1_3_1_37_2","first-page":"449","volume-title":"Proc. 31st Annu. Int. Comput. Softw. Appl. Conf.","author":"Wong W. E.","year":"2007","unstructured":"W. E. Wong, Y. Qi, L. Zhao, and K. Y. Cai. 2007. Effective fault localization using code coverage. In Proc. 31st Annu. Int. Comput. Softw. Appl. Conf., 449\u2013456."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2522920.2522924"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3078840"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.27"},{"key":"e_1_3_1_41_2","unstructured":"Code Review Metrics. Open Web Application Security Project. Retrieved October 9 2015 from https:\/\/owasp.org\/www-project-code-review-guide\/assets\/OWASP_Code_Review_Guide_v2.pdf"},{"key":"e_1_3_1_42_2","unstructured":"Best Practices for Peer Code Review. Smart Bear. Smart Bear Software. Retrieved October 9 2015 from https:\/\/smartbear.com\/learn\/code-review\/best-practices-for-peer-code-review\/"},{"key":"e_1_3_1_43_2","unstructured":"J. G. Ganssle. 2001. A guide to code inspections. White Paper. Retrieved from https:\/\/web.engr.oregonstate.edu\/traylor\/ece473\/pdfs\/ganssel_code_Inspections.pdf"},{"key":"e_1_3_1_44_2","first-page":"69","volume-title":"Proc. 6th IEEE Int. Workshop on Metamorphic Testing","author":"Zhou Z.","year":"2021","unstructured":"Z. Zhou, Z. Zheng, T. Y. Chen, J. Zhou, and K. Qiu. 2021. Follow-up test cases are better than source test cases in metamorphic testing: A preliminary study. In Proc. 6th IEEE Int. Workshop on Metamorphic Testing, 69\u201374."},{"issue":"4","key":"e_1_3_1_45_2","first-page":"437","article-title":"Case studies on testing with compositional metamorphic relations","volume":"24","author":"Dong G.","year":"2008","unstructured":"G. Dong, B. Xu, L. Chen, C. Nie, and L. Wang. 2008. Case studies on testing with compositional metamorphic relations. J. Southeast Univ. (English Edition) 24, 4 (2008), 437\u2013443.","journal-title":"J. Southeast Univ. (English Edition)"},{"key":"e_1_3_1_46_2","volume-title":"Metamorphic Testing Technology for Error Detection Efficiency","author":"Dong G.","year":"2009","unstructured":"G. Dong. 2009. Metamorphic Testing Technology for Error Detection Efficiency. Ph.D. Dissertation. Southeast University."},{"key":"e_1_3_1_47_2","unstructured":"SIR. 2005. Retrieved from http:\/\/sir.unl.edu\/php\/index.php"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111091"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9083-7"},{"key":"e_1_3_1_50_2","unstructured":"FAILTIM Repository. 2022. Retrieved from https:\/\/github.com\/BUAA-SUT\/FAILTIM"},{"key":"e_1_3_1_51_2","volume-title":"On the Analysis of Spectrum-Based Fault Localization","author":"Xie X.","year":"2012","unstructured":"X. Xie. 2012. On the Analysis of Spectrum-Based Fault Localization. Ph.D. Dissertation. Swinburne University of Technology."},{"key":"e_1_3_1_52_2","volume-title":"Research on Metamorphic Testing Technology","author":"Hui Z.","year":"2015","unstructured":"Z. Hui. 2015. Research on Metamorphic Testing Technology. Ph.D. Dissertation. Army Engineering University PLA."},{"key":"e_1_3_1_53_2","first-page":"19","volume-title":"Proc. 29th Annu. Int. Comput. Softw. Appl. Conf.","author":"Wu P.","year":"2005","unstructured":"P. Wu. 2005. Iterative metamorphic testing. In Proc. 29th Annu. Int. Comput. Softw. Appl. Conf., 19\u201324."},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1504\/ijwgs.2020.110945"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2024.3370740"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/227607.227610"},{"key":"e_1_3_1_58_2","first-page":"351","volume-title":"Proc. 30th Int. Conf. Softw. Eng","author":"Namin A. S.","year":"2008","unstructured":"A. S. Namin, J. H. Andrews, and D. J. Murdoch. 2008. Sufficient mutation operators for measuring test effectiveness. Proc. 30th Int. Conf. Softw. Eng. (2008), 351\u2013360."},{"key":"e_1_3_1_59_2","first-page":"90","volume-title":"Proc. 28th Int. Symp. Softw. Reliab. Eng.","author":"Hui Z.","year":"2017","unstructured":"Z. Hui, S. Hong, T. Y. Chen, M. F. Lau, and S. Ng. 2017. Identifying failed test cases through metamorphic testing. In Proc. 28th Int. Symp. Softw. Reliab. Eng., 90\u201391."},{"issue":"3","key":"e_1_3_1_60_2","doi-asserted-by":"crossref","first-page":"1001","DOI":"10.1109\/TSE.2020.3009698","article-title":"Theoretical and empirical analyses of the effectiveness of metamorphic relation composition","volume":"48","author":"Qiu K.","year":"2020","unstructured":"K. Qiu, Z. Zheng, T. Y. Chen, and P. L. Poon. 2020. Theoretical and empirical analyses of the effectiveness of metamorphic relation composition. IEEE Trans. Softw. Eng. 48, 3 (2020), 1001\u20131017.","journal-title":"IEEE Trans. Softw. Eng"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.04.002"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695990","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3695990","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:30Z","timestamp":1750291470000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695990"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,18]]},"references-count":60,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2,28]]}},"alternative-id":["10.1145\/3695990"],"URL":"https:\/\/doi.org\/10.1145\/3695990","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,18]]},"assertion":[{"value":"2023-11-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}