{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T17:02:55Z","timestamp":1784307775170,"version":"3.55.0"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T00:00:00Z","timestamp":1784246400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Italian MUR PRIN 2022 Project","award":["#2022TSYYKJ"],"award-info":[{"award-number":["#2022TSYYKJ"]}]},{"name":"NextGenEu"},{"name":"SERICS","award":["PE00000014"],"award-info":[{"award-number":["PE00000014"]}]},{"DOI":"10.13039\/100031478","name":"NextGenerationEU","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100031478","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Italian Research Group: INdAM-GNCS"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,8,31]]},"abstract":"<jats:p>\n                    In software testing, test suite augmentation refers to the activity of adding new test cases to an existing test suite to enhance its effectiveness, particularly in terms of code coverage and fault detection. In this work, we address the problem of test suite augmentation and propose to leverage implementation dependent test cases that are generated from the coverage of equivalent programs. Specifically, we define\n                    <jats:italic toggle=\"yes\">cross-coverage testing<\/jats:italic>\n                    and\n                    <jats:italic toggle=\"yes\">cross-coverage test suite augmentation<\/jats:italic>\n                    as well as their reference scenarios. Also, we empirically investigate\n                    <jats:italic toggle=\"yes\">cross-coverage test suite augmentation<\/jats:italic>\n                    against two different subjects: a reference benchmark of functionally equivalent programs, and a real-world case study in the context of XACML-based access control systems. Our results show an improvement of the thoroughness of our cross-coverage augmented test suite, as well as of its fault-detection effectiveness, outperforming existing test augmentation strategies.\n                  <\/jats:p>","DOI":"10.1145\/3778362","type":"journal-article","created":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T06:56:59Z","timestamp":1764572219000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Leveraging Cross-Coverage of Equivalent Programs for Test Suite Augmentation"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8749-1356","authenticated-orcid":false,"given":"Antonia","family":"Bertolino","sequence":"first","affiliation":[{"name":"GSSI, L\u2019Aquila, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1076-0076","authenticated-orcid":false,"given":"Guglielmo","family":"De Angelis","sequence":"additional","affiliation":[{"name":"IASI-CNR, Rome, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8760-7299","authenticated-orcid":false,"given":"Felicita Di","family":"Giandomenico","sequence":"additional","affiliation":[{"name":"ISTI-CNR, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4864-2219","authenticated-orcid":false,"given":"Francesca","family":"Lonetti","sequence":"additional","affiliation":[{"name":"ISTI-CNR, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062530"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1985.231893"},{"key":"e_1_3_2_4_2","first-page":"23","article-title":"The methodology of N-version programming","volume":"3","author":"Avizienis Algirdas","year":"1995","unstructured":"Algirdas Avizienis. 1995. The methodology of N-version programming. Software Fault Tolerance 3 (1995), 23\u201346.","journal-title":"Software Fault Tolerance"},{"key":"e_1_3_2_5_2","first-page":"28","volume-title":"Proceedings of the IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops","author":"Bertolino Antonia","year":"2013","unstructured":"Antonia Bertolino, Said Daoudagh, Francesca Lonetti, and Eda Marchetti. 2013. XACMUT: XACML 2.0 mutants generator. In Proceedings of the IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 28\u201333."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1049\/iet-sen.2012.0101"},{"key":"e_1_3_2_7_2","first-page":"101","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automation of Software Test","author":"Bertolino Antonia","year":"2023","unstructured":"Antonia Bertolino, Guglielmo De Angelis, Felicita Di Giandomenico, and Francesca Lonetti. 2023. Cross-coverage testing of functionally equivalent programs. In Proceedings of the IEEE\/ACM International Conference on Automation of Software Test. IEEE, 101\u2013111."},{"key":"e_1_3_2_8_2","first-page":"13","volume-title":"Proceedings of the WEBIST","author":"Bertolino Antonia","year":"2011","unstructured":"Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2011. (role)CAST: A framework for on-line service testing. In Proceedings of the WEBIST, Jos\u00e9 Cordeiro and Joaquim Filipe (Eds.). SciTePress, 13\u201318."},{"key":"e_1_3_2_9_2","first-page":"3","volume-title":"Proceedings of the 36th EUROMICRO Conference on Software Engineering and Advanced Applications","author":"Bertolino Antonia","year":"2010","unstructured":"Antonia Bertolino, Francesca Lonetti, and Eda Marchetti. 2010. Systematic XACML request generation for testing purposes. In Proceedings of the 36th EUROMICRO Conference on Software Engineering and Advanced Applications. IEEE, 3\u201311."},{"key":"e_1_3_2_10_2","first-page":"67","volume-title":"Proceedings of the QRS","author":"Bloem Roderick","year":"2014","unstructured":"Roderick Bloem, Robert Koenighofer, Franz R\u00f6ck, and Michael Tautschnig. 2014. Automating test-suite augmentation. In Proceedings of the QRS. IEEE, 67\u201372."},{"key":"e_1_3_2_11_2","first-page":"334","volume-title":"Proceedings of the ICSME","author":"Borges Hudson","year":"2016","unstructured":"Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Understanding the factors that impact the popularity of GitHub repositories. In Proceedings of the ICSME. IEEE, 334\u2013344."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM59687.2023.00032"},{"key":"e_1_3_2_13_2","first-page":"782","volume-title":"Proceedings of the ICSE","author":"Carzaniga Antonio","year":"2013","unstructured":"Antonio Carzaniga, Alessandra Gorla, Andrea Mattavelli, Nicolo Perino, and Mauro Pezz\u00e8. 2013. Automatic recovery from runtime failures. In Proceedings of the ICSE. IEEE, 782\u2013791."},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1145\/2931037.2948707","volume-title":"Proceedings of the 25th International Symposium on Software Testing and Analysis","author":"Coles Henry","year":"2016","unstructured":"Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A practical mutation testing tool for Java. In Proceedings of the 25th International Symposium on Software Testing and Analysis, 449\u2013452."},{"key":"e_1_3_2_15_2","first-page":"237","volume-title":"Proceedings of the International Symposium on Mathematical Foundations of Computer Science","author":"Cousineau Guy","year":"1979","unstructured":"Guy Cousineau and Patrice Enjalbert. 1979. Program equivalence and provability. In Proceedings of the International Symposium on Mathematical Foundations of Computer Science. Springer, 237\u2013245."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.110398"},{"issue":"1","key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/s11219-019-09470-5","article-title":"XACMET: XACML testing & modeling: An automated model-based testing solution for access control systems","volume":"28","author":"Daoudagh Said","year":"2020","unstructured":"Said Daoudagh, Francesca Lonetti, and Eda Marchetti. 2020. XACMET: XACML testing & modeling: An automated model-based testing solution for access control systems. Software Quality Journal 28, 1 (2020), 249\u2013282.","journal-title":"Software Quality Journal"},{"issue":"3","key":"e_1_3_2_18_2","first-page":"e2306","article-title":"An automated framework for continuous development and testing of access control systems","volume":"35","author":"Daoudagh Said","year":"2023","unstructured":"Said Daoudagh, Francesca Lonetti, and Eda Marchetti. 2023. An automated framework for continuous development and testing of access control systems. Journal of Software: Evolution and Process 35, 3 (2023), e2306.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111674"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.6194"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_3_2_22_2","first-page":"92","volume-title":"Proceedings of the ICSME","author":"Gaston Daniel","year":"2020","unstructured":"Daniel Gaston and James Clause. 2020. A method for finding missing unit tests. In Proceedings of the ICSME. IEEE, 92\u2013103."},{"key":"e_1_3_2_23_2","first-page":"682","volume-title":"Proceedings of the MSR","author":"Higo Yoshiki","year":"2022","unstructured":"Yoshiki Higo, Shinsuke Matsumoto, Shinji Kusumoto, and Kazuya Yasuda. 2022. Constructing dataset of functionally equivalent Java methods using automated test generation techniques. In Proceedings of the MSR, 682\u2013686."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2015.33"},{"key":"e_1_3_2_25_2","first-page":"435","volume-title":"Proceedings of the ICSE","author":"Inozemtseva Laura","year":"2014","unstructured":"Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the ICSE, 435\u2013445."},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1007\/978-3-030-99429-7_11","volume-title":"Proceedings of the International Conference on Fundamental Approaches to Software Engineering","author":"Jakobs Marie-Christine","year":"2022","unstructured":"Marie-Christine Jakobs and Maik Wiesner. 2022. PEQtest: Testing functional equivalence. In Proceedings of the International Conference on Fundamental Approaches to Software Engineering. Springer, Cham, Switzerland, 184\u2013204."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_28_2","first-page":"81","volume-title":"Proceedings of the ISSTA","author":"Jiang Lingxiao","year":"2009","unstructured":"Lingxiao Jiang and Zhendong Su. 2009. Automatic mining of functionally equivalent code fragments via random testing. In Proceedings of the ISSTA, 81\u201392."},{"key":"e_1_3_2_29_2","first-page":"35","volume-title":"Proceedings of the A-TEST","author":"Kessel Marcus","year":"2019","unstructured":"Marcus Kessel and Colin Atkinson. 2019. A platform for diversity-driven test amplification. In Proceedings of the A-TEST, 35\u201341."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111442"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9582-5"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/2.56851"},{"issue":"4","key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1016\/j.infsof.2010.11.009","article-title":"A test-driven approach to code search and its application to the reuse of auxiliary functionality","volume":"53","author":"Lemos Ot\u00e1vio Augusto Lazzarini","year":"2011","unstructured":"Ot\u00e1vio Augusto Lazzarini Lemos, Sushil Bajracharya, Joel Ossher, Paulo Cesar Masiero, and Cristina Lopes. 2011. A test-driven approach to code search and its application to the reuse of auxiliary functionality. Information and Software Technology 53, 4 (2011), 294\u2013306.","journal-title":"Information and Software Technology"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/24.370230"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1145\/3180155.3180210","volume-title":"Proceedings of the 40th International Conference on Software Engineering","author":"Miranda Breno","year":"2018","unstructured":"Breno Miranda, Emilio Cruciani, Roberto Verdecchia, and Antonia Bertolino. 2018. FAST approaches to scalable similarity-based test case prioritization. In Proceedings of the 40th International Conference on Software Engineering, 222\u2013232."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-012-0609-z"},{"key":"e_1_3_2_37_2","unstructured":"OASIS. 2005. XACML 2.0 Conformance Tests v0.4. Retrieved from https:\/\/www.oasis-open.org\/committees\/document.php?document_id=14846"},{"key":"e_1_3_2_38_2","unstructured":"OASIS. 2013. eXtensible Access Control Markup Language (XACML) v3.0. Retrieved from https:\/\/www.oasis-open.org\/standard\/xacmlv3-0\/"},{"key":"e_1_3_2_39_2","first-page":"226","volume-title":"Proceedings of the ASE","author":"Osei-Owusu Jonathan","year":"2019","unstructured":"Jonathan Osei-Owusu, Angello Astorga, Liia Butler, Tao Xie, and Geoffrey Challen. 2019. Grading-based test suite augmentation. In Proceedings of the ASE. IEEE, 226\u2013229."},{"key":"e_1_3_2_40_2","first-page":"75","volume-title":"Proceedings of the of ICSE","author":"Pacheco Carlos","year":"2007","unstructured":"Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the of ICSE, 75\u201384."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/1453101.1453131"},{"key":"e_1_3_2_42_2","first-page":"11","volume-title":"Proceedings of the ICST","author":"Pezz\u00e8 Mauro","year":"2013","unstructured":"Mauro Pezz\u00e8, Konstantin Rubinov, and Jochen Wuttke. 2013. Generating effective integration test cases from unit ones. In Proceedings of the ICST. IEEE, 11\u201320."},{"key":"e_1_3_2_43_2","first-page":"355","volume-title":"Proceedings of the 36th IEEE\/ACM International Conference on Automated Software Engineering.","author":"Riccio Vincenzo","year":"2021","unstructured":"Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. DeepMetis: Augmenting a deep learning test set to increase its mutation score. In Proceedings of the 36th IEEE\/ACM International Conference on Automated Software Engineering. IEEE, 355\u2013367."},{"key":"e_1_3_2_44_2","first-page":"218","volume-title":"Proceedings of theASE","author":"Santelices Raul","year":"2008","unstructured":"Raul Santelices, Pavan Kumar Chittimalli, Taweesup Apiwattanapong, Alessandro Orso, and Mary Jean Harrold. 2008. Test-suite augmentation for evolving software. In Proceedings of the ASE. IEEE, 218\u2013227."},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1145\/3677995.3678194","volume-title":"Proceedings of the 23rd ACM SIGPLAN International Workshop on Erlang","author":"Seres Bendeg\u00faz","year":"2024","unstructured":"Bendeg\u00faz Seres, D\u00e1niel Horp\u00e1csi, and Simon Thompson. 2024. Is this really a refactoring? Automated equivalence checking for erlang projects. In Proceedings of the 23rd ACM SIGPLAN International Workshop on Erlang, 55\u201366."},{"key":"e_1_3_2_46_2","first-page":"1","volume-title":"Proceedings of theICIIS","author":"Sikri Monika","year":"2014","unstructured":"Monika Sikri and Rajkumar Gell. 2014. Adaptive SOA stack and discovery framework for redundant services. In Proceedings of the ICIIS. IEEE, 1\u20136."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9083-7"},{"key":"e_1_3_2_48_2","volume-title":"Software Diversity in Computerized Control Systems","author":"Voges Udo","year":"2012","unstructured":"Udo Voges. 2012. Software Diversity in Computerized Control Systems. Vol. 2. Springer Science & Business Media."},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/CMPASS.1988.9641","volume-title":"Proceedings of the Computer Assurance, 1988 (COMPASS\u201988)","author":"Vouk Mladen A.","year":"1988","unstructured":"Mladen A. Vouk. 1988. On back-to-back testing. In Proceedings of the Computer Assurance, 1988 (COMPASS\u201988). IEEE, 84\u201391."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560426"},{"key":"e_1_3_2_51_2","first-page":"1365","volume-title":"Proceedings of the of GECCO","author":"Xu Zhihong","year":"2010","unstructured":"Zhihong Xu, Myra B. Cohen, and Gregg Rothermel. 2010. Factors affecting the use of genetic algorithms in test suite augmentation. In Proceedings of the of GECCO, 1365\u20131372."},{"key":"e_1_3_2_52_2","first-page":"150","volume-title":"Proceedings of the ISSRE","author":"Xu Zhihong","year":"2011","unstructured":"Zhihong Xu, Yunho Kim, Moonzoo Kim, and Gregg Rothermel. 2011. A hybrid directed test suite augmentation technique. In Proceedings of the ISSRE. IEEE, 150\u2013159."},{"key":"e_1_3_2_53_2","first-page":"257","volume-title":"Proceedings of the FSE","author":"Xu Zhihong","year":"2010","unstructured":"Zhihong Xu, Yunho Kim, Moonzoo Kim, Gregg Rothermel, and Myra B. Cohen. 2010. Directed test suite augmentation: Techniques and tradeoffs. In Proceedings of the FSE, 257\u2013266."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.435"},{"issue":"4","key":"e_1_3_2_55_2","first-page":"405","article-title":"ConTesa: Directed test suite augmentation for concurrent software","volume":"46","author":"Yu Tingting","year":"2018","unstructured":"Tingting Yu, Zunchen Huang, and Chao Wang. 2018. ConTesa: Directed test suite augmentation for concurrent software. IEEE Transactions on Software Engineering 46, 4 (2018), 405\u2013419.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3778362","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T16:04:03Z","timestamp":1784304243000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3778362"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,17]]},"references-count":55,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2026,8,31]]}},"alternative-id":["10.1145\/3778362"],"URL":"https:\/\/doi.org\/10.1145\/3778362","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,17]]},"assertion":[{"value":"2025-07-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-13","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}