{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T14:33:27Z","timestamp":1754145207220,"version":"3.41.2"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","license":[{"start":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T00:00:00Z","timestamp":1750550400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CCF-2008905,CCF-2047682"],"award-info":[{"award-number":["CCF-2008905,CCF-2047682"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Fraunhofer Internal Programs","award":["PREPARE 840 231"],"award-info":[{"award-number":["PREPARE 840 231"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Fuzzing is a powerful software testing technique renowned for its effectiveness in identifying software vulnerabilities. Traditional fuzzing evaluations typically focus on overall fuzzer performance across a set of target programs, yet few benchmarks consider how fine-grained program features influence fuzzing effectiveness. To bridge this gap, we introduce FeatureBench, a novel benchmark designed to generate programs with configurable, fine-grained program features to enhance fuzzing evaluations. We reviewed 25 recent grey-box fuzzing studies, extracting 7 program features related to control-flow and data-flow that can impact fuzzer performance. Using these features, we generated a benchmark consisting of 153 programs controlled by 10 fine-grained configurable parameters. We evaluated 11 fuzzers using this benchmark, with each fuzzer representing either distinct claimed improvements or serving as a widely used baseline in fuzzing evaluations. The results indicate that fuzzer performance varies significantly based on the program features and their strengths, highlighting the importance of incorporating program characteristics into fuzzing evaluations.<\/jats:p>","DOI":"10.1145\/3728899","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"527-549","source":"Crossref","is-referenced-by-count":0,"title":["Program Feature-Based Benchmarking for Fuzz Testing"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-6650-6239","authenticated-orcid":false,"given":"Miao","family":"Miao","sequence":"first","affiliation":[{"name":"University of Texas at Dallas, Richardson, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1963-4265","authenticated-orcid":false,"given":"Sriteja","family":"Kummita","sequence":"additional","affiliation":[{"name":"Fraunhofer IEM, Paderborn, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3470-3647","authenticated-orcid":false,"given":"Eric","family":"Bodden","sequence":"additional","affiliation":[{"name":"Heinz Nixdorf Institute at Paderborn University, Paderborn, Germany"},{"name":"Fraunhofer IEM, Paderborn, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2826-1857","authenticated-orcid":false,"given":"Shiyi","family":"Wei","sequence":"additional","affiliation":[{"name":"University of Texas at Dallas, Richardson, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Damien Octeau, and Patrick McDaniel.","author":"Arzt Steven","year":"2014","unstructured":"Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. ACM sigplan notices, 49, 6 (2014), 259\u2013269."},{"key":"e_1_2_1_2_1","first-page":"1","article-title":"REDQUEEN: Fuzzing with Input-to-State Correspondence","volume":"19","author":"Aschermann Cornelius","year":"2019","unstructured":"Cornelius Aschermann, Sergej Schumilo, Tim Blazytko, Robert Gawlik, and Thorsten Holz. 2019. REDQUEEN: Fuzzing with Input-to-State Correspondence.. In NDSS. 19, 1\u201315.","journal-title":"NDSS."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978428"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510230"},{"key":"e_1_2_1_5_1","unstructured":"DARPA CGC. 2018. Darpa Cyber Grand Challenge (CGC). https:\/\/github.com\/CyberGrandChallenge\/"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00046"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.15"},{"key":"e_1_2_1_8_1","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 60\u201371","author":"Du Xiaoning","year":"2019","unstructured":"Xiaoning Du, Bihuan Chen, Yuekang Li, Jianmin Guo, Yaqin Zhou, Yang Liu, and Yu Jiang. 2019. Leopard: Identifying vulnerable code for vulnerability assessment through program metrics. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 60\u201371."},{"key":"e_1_2_1_9_1","volume-title":"14th USENIX Workshop on Offensive Technologies (WOOT 20)","author":"Fioraldi Andrea","year":"2020","unstructured":"Andrea Fioraldi, Dominik Maier, Heiko Ei\u00df feldt, and Marc Heuse. 2020. $AFL++$: Combining incremental steps of fuzzing research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20)."},{"key":"e_1_2_1_10_1","volume-title":"Dissecting american fuzzy lop: a fuzzbench evaluation. ACM transactions on software engineering and methodology, 32, 2","author":"Fioraldi Andrea","year":"2023","unstructured":"Andrea Fioraldi, Alessandro Mantovani, Dominik Maier, and Davide Balzarotti. 2023. Dissecting american fuzzy lop: a fuzzbench evaluation. ACM transactions on software engineering and methodology, 32, 2 (2023), 1\u201326."},{"key":"e_1_2_1_11_1","unstructured":"FuzzBench. 2020. FuzzBench: 2020-09-07 report. https:\/\/www.fuzzbench.com\/reports\/sample\/index.html"},{"key":"e_1_2_1_12_1","unstructured":"Google. 2014. Honggfuzz. https:\/\/github.com\/google\/honggfuzz"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428334"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-021-00617-z"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP40000.2020.00063"},{"key":"e_1_2_1_16_1","volume-title":"DARWIN: Survival of the fittest fuzzing mutators. arXiv preprint arXiv:2210.11783.","author":"Jauernig Patrick","year":"2022","unstructured":"Patrick Jauernig, Domagoj Jakobovic, Stjepan Picek, Emmanuel Stapf, and Ahmad-Reza Sadeghi. 2022. DARWIN: Survival of the fittest fuzzing mutators. arXiv preprint arXiv:2210.11783."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3678722.3685530"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCQ51190.2021.9392986"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695284"},{"key":"e_1_2_1_21_1","unstructured":"Laf-intel. 2016. Circumventing Fuzzing Roadblocks with Compiler Transformations. https:\/\/lafintel.wordpress.com\/"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CGO.2004.1281665"},{"key":"e_1_2_1_23_1","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 384\u2013396","author":"Lee Myungho","year":"2023","unstructured":"Myungho Lee, Sooyoung Cha, and Hakjoo Oh. 2023. Learning seed-adaptive mutation strategies for greybox fuzzing. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 384\u2013396."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238176"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106295"},{"key":"e_1_2_1_26_1","volume-title":"30th USENIX Security Symposium (USENIX Security 21)","author":"Li Yuwei","year":"2021","unstructured":"Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, and Peng Cheng. 2021. $UNIFUZZ$: A holistic and pragmatic $Metrics-Driven$ platform for evaluating fuzzers. In 30th USENIX Security Symposium (USENIX Security 21). 2777\u20132794."},{"key":"e_1_2_1_27_1","volume-title":"2022 IEEE Symposium on Security and Privacy (SP). 1\u201317","author":"Liang Jie","year":"2022","unstructured":"Jie Liang, Mingzhe Wang, Chijin Zhou, Zhiyong Wu, Yu Jiang, Jianzhong Liu, Zhe Liu, and Jiaguang Sun. 2022. Pata: Fuzzing with path aware taint analysis. In 2022 IEEE Symposium on Security and Privacy (SP). 1\u201317."},{"key":"e_1_2_1_28_1","unstructured":"LLVM. 2015. LibFuzzer. https:\/\/llvm.org\/docs\/LibFuzzer.html"},{"key":"e_1_2_1_29_1","volume-title":"28th USENIX Security Symposium (USENIX Security 19)","author":"Lyu Chenyang","year":"2019","unstructured":"Chenyang Lyu, Shouling Ji, Chao Zhang, Yuwei Li, Wei-Han Lee, Yu Song, and Raheem Beyah. 2019. $MOPT$: Optimized mutation scheduling for fuzzers. In 28th USENIX Security Symposium (USENIX Security 19). 1949\u20131966."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534385"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics 50\u201360.","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473932"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","unstructured":"Miao Miao Sriteja Kummita Eric Bodden and Shiyi Wei. 2025. Artifacts for the paper \"Program Feature-based Benchmarking for Fuzz Testing\". https:\/\/doi.org\/10.5281\/zenodo.15200822 10.5281\/zenodo.15200822","DOI":"10.5281\/zenodo.15200822"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1396\u20131406","author":"Ounjai Jiradet","year":"2023","unstructured":"Jiradet Ounjai, Valentin W\u00fcstholz, and Maria Christakis. 2023. Green Fuzzer Benchmarking. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1396\u20131406."},{"key":"e_1_2_1_35_1","volume-title":"Computer Security\u2013ESORICS 2021: 26th European Symposium on Research in Computer Security","author":"David Paa\u00df","year":"2021","unstructured":"David Paa\u00df en, Sebastian Surminski, Michael Rodler, and Lucas Davi. 2021. My fuzzer beats them all! developing a framework for fair evaluation and comparison of fuzzers. In Computer Security\u2013ESORICS 2021: 26th European Symposium on Research in Computer Security, Darmstadt, Germany, October 4\u20138, 2021, Proceedings, Part I 26. 173\u2013193."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00056"},{"key":"e_1_2_1_37_1","unstructured":"Mohit Rajpal William Blum and Rishabh Singh. 2017. Not all bytes are equal: Neural byte sieve for fuzzing. arXiv preprint arXiv:1711.04596."},{"key":"e_1_2_1_38_1","first-page":"1","article-title":"VUzzer: Application-aware evolutionary fuzzing","volume":"17","author":"Rawat Sanjay","year":"2017","unstructured":"Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida, and Herbert Bos. 2017. VUzzer: Application-aware evolutionary fuzzing.. In NDSS. 17, 1\u201314.","journal-title":"NDSS."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330555"},{"key":"e_1_2_1_40_1","unstructured":"RERS. 2022. The RERS Challenge. https:\/\/rers-challenge.org\/"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598136"},{"key":"e_1_2_1_42_1","volume-title":"2024 IEEE Symposium on Security and Privacy (SP). 1974\u20131993","author":"Schloegel Moritz","year":"2024","unstructured":"Moritz Schloegel, Nils Bars, Nico Schiller, Lukas Bernhard, Tobias Scharnowski, Addison Crump, Arash Ale-Ebrahim, Nicolai Bissantz, Marius Muench, and Thorsten Holz. 2024. Sok: Prudent evaluation practices for fuzzing. In 2024 IEEE Symposium on Security and Privacy (SP). 1974\u20131993."},{"volume-title":"OSS-Fuzz - Google","author":"Serebryany Kostya","key":"e_1_2_1_43_1","unstructured":"Kostya Serebryany. 2017. OSS-Fuzz - Google\u2019 s continuous fuzzing service for open source software. USENIX Association, Vancouver, BC."},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","unstructured":"Charles Spearman. 1961. The proof and measurement of association between two things..","DOI":"10.1037\/11491-005"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2010.37"},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Yanhao Wang Xiangkun Jia Yuwei Liu Kyle Zeng Tiffany Bao Dinghao Wu and Purui Su. 2020. Not All Coverage Measurements Are Equal: Fuzzing by Coverage Accounting for Input Prioritization.. In NDSS.","DOI":"10.14722\/ndss.2020.24422"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380396"},{"key":"e_1_2_1_48_1","unstructured":"Dylan Wolff Marcel B\u00f6hme and Abhik Roychoudhury. 2022. Explainable fuzzer evaluation. arXiv preprint arXiv:2212.09519."},{"key":"e_1_2_1_49_1","volume-title":"29th USENIX Security Symposium (USENIX Security 20)","author":"Yue Tai","year":"2020","unstructured":"Tai Yue, Pengfei Wang, Yong Tang, Enze Wang, Bo Yu, Kai Lu, and Xu Zhou. 2020. $EcoFuzz$: Adaptive $Energy-Saving$ greybox fuzzing as a variant of the adversarial $Multi-Armed$ bandit. In 29th USENIX Security Symposium (USENIX Security 20). 2307\u20132324."},{"key":"e_1_2_1_50_1","unstructured":"Micha\u0142 Zalewski. 2013. American Fuzzy Lop (2.52b). https:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_2_1_51_1","volume-title":"Mobfuzz: Adaptive multi-objective optimization in gray-box fuzzing. arXiv preprint arXiv:2401.15956.","author":"Zhang Gen","year":"2024","unstructured":"Gen Zhang, Pengfei Wang, Tai Yue, Xiangdong Kong, Shan Huang, Xu Zhou, and Kai Lu. 2024. Mobfuzz: Adaptive multi-objective optimization in gray-box fuzzing. arXiv preprint arXiv:2401.15956."},{"key":"e_1_2_1_52_1","volume-title":"SHAPFUZZ: Efficient Fuzzing via Shapley-Guided Byte Selection. arXiv preprint arXiv:2308.09239.","author":"Zhang Kunpeng","year":"2023","unstructured":"Kunpeng Zhang, Xiaogang Zhu, Xi Xiao, Minhui Xue, Chao Zhang, and Sheng Wen. 2023. SHAPFUZZ: Efficient Fuzzing via Shapley-Guided Byte Selection. arXiv preprint arXiv:2308.09239."},{"key":"e_1_2_1_53_1","volume-title":"FIXREVERTER: A Realistic Bug Injection Methodology for Benchmarking Fuzz Testing. In 31st USENIX Security Symposium (USENIX Security 22)","author":"Zhang Zenong","year":"2022","unstructured":"Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. 2022. FIXREVERTER: A Realistic Bug Injection Methodology for Benchmarking Fuzz Testing. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA. 3699\u20133715. isbn:978-1-939133-31-1 https:\/\/www.usenix.org\/conference\/usenixsecurity22\/presentation\/zhang-zenong"},{"key":"e_1_2_1_54_1","doi-asserted-by":"crossref","unstructured":"Lei Zhao Yue Duan and Jifeng XUAN. 2019. Send hardest problems my way: Probabilistic path prioritization for hybrid fuzzing.","DOI":"10.14722\/ndss.2019.23504"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.3390\/math9030205"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3321705.3329845"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728899","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728899","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:50:38Z","timestamp":1752684638000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":56,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728899"],"URL":"https:\/\/doi.org\/10.1145\/3728899","relation":{},"ISSN":["2994-970X"],"issn-type":[{"type":"electronic","value":"2994-970X"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}