{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:03:26Z","timestamp":1750309406281,"version":"3.41.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T00:00:00Z","timestamp":1732924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>To foster the verifiability and testability of deep neural networks (DNN), an increasing number of methods for test case generation techniques are being developed.<\/jats:p>\n          <jats:p>When confronted with testing DNN models, the user can apply any existing test generation technique. However, it needs to do so for each technique and each DNN model under test, which can be expensive. Therefore, a paradigm shift could benefit this testing process: rather than regenerating the test set independently for each DNN model under test, we could transfer from existing DNN models.<\/jats:p>\n          <jats:p>This article introduces Generated Inputs Sets Transferability (GIST), a novel approach for the efficient transfer of test sets. Given a property selected by a user (e.g., neurons covered, faults), GIST enables the selection of good test sets from the point of view of this property among available test sets. This allows the user to recover similar properties on the transferred test sets as he would have obtained by generating the test set from scratch with a test cases generation technique. Experimental results show that GIST can select effective test sets for the given property to transfer. Moreover, GIST scales better than reapplying test case generation techniques from scratch on DNN models under test.<\/jats:p>","DOI":"10.1145\/3672457","type":"journal-article","created":{"date-parts":[[2024,6,13]],"date-time":"2024-06-13T17:56:59Z","timestamp":1718301419000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["<i>GIST<\/i>\n            : Generated Inputs Sets Transferability in Deep Learning"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5593-9400","authenticated-orcid":false,"given":"Florian","family":"Tambon","sequence":"first","affiliation":[{"name":"Polytechnique Montreal, Montreal, Quebec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5704-4173","authenticated-orcid":false,"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"Polytechnique Montreal, Montreal, Quebec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7892-9202","authenticated-orcid":false,"given":"Giuliano","family":"Antoniol","sequence":"additional","affiliation":[{"name":"Polytechnique Montreal, Montreal, Quebec, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,12,3]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2023. HuggingFace. Retrieved from https:\/\/huggingface.co\/"},{"key":"e_1_3_2_3_2","unstructured":"2023. ReplicationPackage. Retrieved from https:\/\/github.com\/FlowSs\/GIST or https:\/\/zenodo.org\/records\/10028594"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1111\/opo.12131"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550271"},{"key":"e_1_3_2_8_2","first-page":"7115","article-title":"GULP: A prediction-based metric between representations","volume":"35","author":"Boix-Adsera Enric","year":"2022","unstructured":"Enric Boix-Adsera, Hannah Lawrence, George Stepaniants, and Philippe Rigollet. 2022. GULP: A prediction-based metric between representations. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35. 7115\u20137127.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_9_2","first-page":"454","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME \u201919)","author":"Braiek Houssem Ben","year":"2019","unstructured":"Houssem Ben Braiek and Foutse Khomh. 2019. DeepEvolution: A search-based testing approach for deep neural networks. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME \u201919). IEEE, 454\u2013458."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542"},{"issue":"2","key":"e_1_3_2_11_2","first-page":"1","article-title":"Comprehensive survey on distance\/similarity measures between probability density functions","volume":"1","author":"Cha Sung-Hyuk","year":"2007","unstructured":"Sung-Hyuk Cha. 2007. Comprehensive survey on distance\/similarity measures between probability density functions. City 1, 2 (2007), 1.","journal-title":"City"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582573"},{"key":"e_1_3_2_13_2","unstructured":"MMGeneration Contributors. 2021. MMGeneration: OpenMMLab Generative Model Toolbox and Benchmark. Retrieved from https:\/\/github.com\/open-mmlab\/mmgeneration"},{"key":"e_1_3_2_14_2","first-page":"5656","article-title":"Similarity and matching of neural network representations","volume":"34","author":"Csisz\u00e1rik Adri\u00e1n","year":"2021","unstructured":"Adri\u00e1n Csisz\u00e1rik, P\u00e9ter K\u0151r\u00f6si-Szab\u00f3, Akos Matszangosz, Gergely Papp, and D\u00e1niel Varga. 2021. Similarity and matching of neural network representations. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 34. 5656\u20135668.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_15_2","first-page":"1556","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Ding Frances","year":"2021","unstructured":"Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. 2021. Grounding representation similarity through statistical testing. In Proceedings of the Advances in Neural Information Processing Systems. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 1556\u20131568. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2021\/file\/0c0bf917c7942b5a08df71f9da626f97-Paper.pdf"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464801"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"877","DOI":"10.1007\/978-3-030-71704-9_65","volume-title":"Advances in Data Science and Information Engineering","author":"Farahani Abolfazl","year":"2021","unstructured":"Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. 2021. A brief review of domain adaptation. In Advances in Data Science and Information Engineering. Robert Stahlbock, Gary M. Weiss, Mahmoud Abou-Nasr, Cheng-Ying Yang, Hamid R. Arabnia, and Leonidas Deligiannidis (Eds.). Springer International Publishing, Cham, 877\u2013894."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_2_22_2","first-page":"630","volume-title":"Proceedings of the Computer Vision\u2013ECCV 2016: 14th European Conference, Part IV 14","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In Proceedings of the Computer Vision\u2013ECCV 2016: 14th European Conference, Part IV 14. Springer, 630\u2013645."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511598"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00723"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2921336"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.3390\/sym11091066"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.5555\/2823861"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","unstructured":"Max Klabunde Tobias Schumacher Markus Strohmaier and Florian Lemmerich. 2023. Similarity of neural network models: A survey of functional and representational measures. arXiv:2305.06329. Retrieved from 10.48550\/arXiv.2305.06329","DOI":"10.48550\/arXiv.2305.06329"},{"key":"e_1_3_2_31_2","first-page":"3519","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Kornblith Simon","year":"2019","unstructured":"Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the International Conference on Machine Learning. PMLR, 3519\u20133529."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.3390\/info10040150"},{"key":"e_1_3_2_33_2","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images . Retrieved from https:\/\/www.cs.toronto.edu\/~kriz\/learning-features-2009-TR.pdf"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403262"},{"key":"e_1_3_2_36_2","unstructured":"Wei Li. 2019. CIFAR-ZOO: PyTorch Implementation of CNNs for CIFAR Dataset. Retrieved from https:\/\/github.com\/BIGBALLON\/CIFAR-ZOO"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464816"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00015"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","unstructured":"Wei Ma Mike Papadakis Anestis Tsakmalis Maxime Cordy and Yves Le Traon. 2021. Test selection for deep learning systems. ACM Transactions on Software Engineering and Methodology 30 2 Article 13 (Jan. 2021) 22 pages. DOI: 10.1145\/3417330","DOI":"10.1145\/3417330"},{"key":"e_1_3_2_42_2","first-page":"873","article-title":"Co-validation: Using model disagreement on unlabeled data to validate classification algorithms","volume":"17","author":"Madani Omid","year":"2004","unstructured":"Omid Madani, David Pennock, and Gary Flake. 2004. Co-validation: Using model disagreement on unlabeled data to validate classification algorithms. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 17. 873\u2013880.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","unstructured":"Leland McInnes John Healy and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426. Retrieved from 10.48550\/arXiv.1802.03426","DOI":"10.48550\/arXiv.1802.03426"},{"key":"e_1_3_2_45_2","first-page":"5732","article-title":"Insights on representational similarity in neural networks with canonical correlation","volume":"31","author":"Morcos Ari","year":"2018","unstructured":"Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 31. 5732\u20135741.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.341"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.16"},{"key":"e_1_3_2_48_2","first-page":"839","volume-title":"Proceedings of the 2014 SIAM International Conference on Data Mining. SIAM","author":"Moulavi Davoud","year":"2014","unstructured":"Davoud Moulavi, Pablo A. Jaskowiak, Ricardo J. G. B. Campello, Arthur Zimek, and J\u00f6rg Sander. 2014. Density-based clustering validation. In Proceedings of the 2014 SIAM International Conference on Data Mining. SIAM, 839\u2013847."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","unstructured":"Anna Ollerenshaw Md Asif Jalal and Thomas Hain. 2022. Insights on neural representations for end-to-end speech recognition. arXiv:2205.09456. Retrieved from 10.48550\/arXiv.2205.09456","DOI":"10.48550\/arXiv.2205.09456"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_3_2_51_2","first-page":"115","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Pang Bo","year":"2005","unstructured":"Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association for Computational Linguistics. 115\u2013124."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2019.2908700"},{"key":"e_1_3_2_54_2","first-page":"12116","article-title":"Do vision transformers see like convolutional neural networks?","volume":"34","author":"Raghu Maithra","year":"2021","unstructured":"Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers see like convolutional neural networks? In Proceedings of the Advances in Neural Information Processing Systems, Vol. 34. 12116\u201312128.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.3390\/info11040193"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409730"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00104"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416621"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-6106"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","unstructured":"Sharath Nittur Sridhar and Anthony Sarah. 2020. Undivided attention: Are intermediate layers necessary for bert? arXiv:2012.11881. Retrieved from 10.48550\/arXiv.2012.11881","DOI":"10.48550\/arXiv.2012.11881"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-022-00337-x"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","unstructured":"Florian Tram\u00e8r Nicolas Papernot Ian Goodfellow Dan Boneh and Patrick McDaniel. 2017. The space of transferable adversarial examples. arXiv:1704.03453. Retrieved from 10.48550\/arXiv.1704.03453","DOI":"10.48550\/arXiv.1704.03453"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/7068349"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.586"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503161.3547989"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00130"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3607190"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672457","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672457","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:01Z","timestamp":1750294681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672457"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,30]]},"references-count":70,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3672457"],"URL":"https:\/\/doi.org\/10.1145\/3672457","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2024,11,30]]},"assertion":[{"value":"2023-10-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}