{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T02:14:15Z","timestamp":1775873655428,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,18]],"date-time":"2021-08-18T00:00:00Z","timestamp":1629244800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,20]]},"DOI":"10.1145\/3468264.3473932","type":"proceedings-article","created":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T01:40:20Z","timestamp":1629337220000},"page":"1393-1403","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":158,"title":["FuzzBench: an open fuzzer benchmarking platform and service"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7042-0444","authenticated-orcid":false,"given":"Jonathan","family":"Metzman","sequence":"first","affiliation":[{"name":"Google, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9808-6484","authenticated-orcid":false,"given":"L\u00e1szl\u00f3","family":"Szekeres","sequence":"additional","affiliation":[{"name":"Google, USA"}]},{"given":"Laurent","family":"Simon","sequence":"additional","affiliation":[{"name":"Google, USA"}]},{"given":"Read","family":"Sprabery","sequence":"additional","affiliation":[{"name":"Google, USA"}]},{"given":"Abhishek","family":"Arya","sequence":"additional","affiliation":[{"name":"Google, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,8,18]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2021. OSS-Fuzz Bugs. https:\/\/bugs.chromium.org\/p\/oss-fuzz\/issues\/list?q=-status  2021. OSS-Fuzz Bugs. https:\/\/bugs.chromium.org\/p\/oss-fuzz\/issues\/list?q=-status"},{"key":"e_1_3_2_1_2_1","unstructured":"ACM SIGPLAN. 2018. Empirical Evaluation Guidelines. http:\/\/www.sigplan.org\/Resources\/EmpiricalEvaluation\/  ACM SIGPLAN. 2018. Empirical Evaluation Guidelines. http:\/\/www.sigplan.org\/Resources\/EmpiricalEvaluation\/"},{"key":"e_1_3_2_1_3_1","volume-title":"Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software. Google Testing Blog. (Dec","author":"Aizatsky Mike","year":"2016","unstructured":"Mike Aizatsky , Kostya Serebryany , Oliver Chang , Abhishek Arya , and Meredith Whittaker . 2016. Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software. Google Testing Blog. (Dec . 2016 ). https:\/\/testing.googleblog.com\/2016\/12\/announcing-oss-fuzz-continuous-fuzzing.html Mike Aizatsky, Kostya Serebryany, Oliver Chang, Abhishek Arya, and Meredith Whittaker. 2016. Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software. Google Testing Blog. (Dec. 2016). https:\/\/testing.googleblog.com\/2016\/12\/announcing-oss-fuzz-continuous-fuzzing.html"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.121"},{"key":"e_1_3_2_1_5_1","unstructured":"Abhishek Arya and Chang Oliver. 2019. ClusterFuzz: ClusterFuzz. https:\/\/www.blackhat.com\/eu-19\/briefings\/schedule\/index.html#clusterfuzz-fuzzing-at-google-scale-17505 Black Hat Europe.  Abhishek Arya and Chang Oliver. 2019. ClusterFuzz: ClusterFuzz. https:\/\/www.blackhat.com\/eu-19\/briefings\/schedule\/index.html#clusterfuzz-fuzzing-at-google-scale-17505 Black Hat Europe."},{"key":"e_1_3_2_1_6_1","first-page":"1","article-title":"REDQUEEN: Fuzzing with Input-to-State Correspondence","volume":"19","author":"Aschermann Cornelius","year":"2019","unstructured":"Cornelius Aschermann , Sergej Schumilo , Tim Blazytko , Robert Gawlik , and Thorsten Holz . 2019 . REDQUEEN: Fuzzing with Input-to-State Correspondence .. In NDSS , Vol. 19. 1 \u2013 15 . Cornelius Aschermann, Sergej Schumilo, Tim Blazytko, Robert Gawlik, and Thorsten Holz. 2019. REDQUEEN: Fuzzing with Input-to-State Correspondence.. In NDSS, Vol. 19. 1\u201315.","journal-title":"NDSS"},{"key":"e_1_3_2_1_7_1","unstructured":"FuzzBench Authors. 2021. FuzzBench. https:\/\/github.com\/google\/fuzzbench\/tree\/d51da57c211af226dd7854085d9fc80070205738\/fuzzers\/symcc_afl.  FuzzBench Authors. 2021. FuzzBench. https:\/\/github.com\/google\/fuzzbench\/tree\/d51da57c211af226dd7854085d9fc80070205738\/fuzzers\/symcc_afl."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Dirk Beyer. 2020. Competition on Software Testing. https:\/\/test-comp.sosy-lab.org\/2020\/  Dirk Beyer. 2020. Competition on Software Testing. https:\/\/test-comp.sosy-lab.org\/2020\/","DOI":"10.1007\/978-3-030-45234-6_25"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409748"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2785841"},{"key":"e_1_3_2_1_11_1","volume-title":"Evaluating Synthetic Bugs. Hong Kong","author":"Bundt Joshua","year":"2021","unstructured":"Joshua Bundt , Andrew Fasano , Brendan Dolan-Gavitt , William Robertson , and Tim Leek . 2021. Evaluating Synthetic Bugs. Hong Kong ( 2021 ), 14. Joshua Bundt, Andrew Fasano, Brendan Dolan-Gavitt, William Robertson, and Tim Leek. 2021. Evaluating Synthetic Bugs. Hong Kong (2021), 14."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00082"},{"key":"e_1_3_2_1_13_1","unstructured":"DARPA. 2016. DARPA Cyber Grand Challenge Final Event Archive. http:\/\/www.lungetech.com\/cgc-corpus\/  DARPA. 2016. DARPA Cyber Grand Challenge Final Event Archive. http:\/\/www.lungetech.com\/cgc-corpus\/"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1248547.1248548"},{"key":"e_1_3_2_1_15_1","unstructured":"Song Dokyung. 2020. Add an option to keep initial seed inputs around. https:\/\/reviews.llvm.org\/D86577  Song Dokyung. 2020. Add an option to keep initial seed inputs around. https:\/\/reviews.llvm.org\/D86577"},{"key":"e_1_3_2_1_16_1","unstructured":"Song Dokyung. 2020. Enable entropic by default. https:\/\/reviews.llvm.org\/D87476  Song Dokyung. 2020. Enable entropic by default. https:\/\/reviews.llvm.org\/D87476"},{"key":"e_1_3_2_1_17_1","unstructured":"Song Dokyung. 2020. Fix arguments of InsertPartOf\/CopyPartOf calls in CrossOver mutator. https:\/\/github.com\/llvm\/llvm-project\/commit\/c10e63677f5d20f18010f8f68c631ddc97546f7d  Song Dokyung. 2020. Fix arguments of InsertPartOf\/CopyPartOf calls in CrossOver mutator. https:\/\/github.com\/llvm\/llvm-project\/commit\/c10e63677f5d20f18010f8f68c631ddc97546f7d"},{"key":"e_1_3_2_1_18_1","unstructured":"Song Dokyung. 2020. Link libFuzzer\u2019s own interceptors when other compiler runtimes are not linked. https:\/\/reviews.llvm.org\/D83494  Song Dokyung. 2020. Link libFuzzer\u2019s own interceptors when other compiler runtimes are not linked. https:\/\/reviews.llvm.org\/D83494"},{"key":"e_1_3_2_1_19_1","unstructured":"Song Dokyung. 2020. Scale energy assigned to each input based on input execution time. https:\/\/github.com\/llvm\/llvm-project\/commit\/5cda4dc7b4d28fcd11307d4234c513ff779a1c6f  Song Dokyung. 2020. Scale energy assigned to each input based on input execution time. https:\/\/github.com\/llvm\/llvm-project\/commit\/5cda4dc7b4d28fcd11307d4234c513ff779a1c6f"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.15"},{"key":"e_1_3_2_1_21_1","unstructured":"Andrew Fasano. 2019. Rode0day: A Year of Bug-Finding Evaluations.  Andrew Fasano. 2019. Rode0day: A Year of Bug-Finding Evaluations."},{"key":"e_1_3_2_1_22_1","volume-title":"14th \\\\USENIX\\ Workshop on Offensive Technologies (\\\\WOOT\\ 20). https:\/\/www.usenix.org\/conference\/woot20\/presentation\/fioraldi","author":"Fioraldi Andrea","unstructured":"Andrea Fioraldi , Dominik Maier , Heiko Ei\u00dffeldt , and Marc Heuse . 2020. AFL++ : Combining Incremental Steps of Fuzzing Research . In 14th \\\\USENIX\\ Workshop on Offensive Technologies (\\\\WOOT\\ 20). https:\/\/www.usenix.org\/conference\/woot20\/presentation\/fioraldi Andrea Fioraldi, Dominik Maier, Heiko Ei\u00dffeldt, and Marc Heuse. 2020. AFL++ : Combining Incremental Steps of Fuzzing Research. In 14th \\\\USENIX\\ Workshop on Offensive Technologies (\\\\WOOT\\ 20). https:\/\/www.usenix.org\/conference\/woot20\/presentation\/fioraldi"},{"key":"e_1_3_2_1_23_1","unstructured":"FuzzBench. 2021. The report \u2013 FuzzBench. https:\/\/google.github.io\/fuzzbench\/reference\/report\/  FuzzBench. 2021. The report \u2013 FuzzBench. https:\/\/google.github.io\/fuzzbench\/reference\/report\/"},{"key":"e_1_3_2_1_24_1","volume-title":"An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March","author":"Geng Sijia","year":"2020","unstructured":"Sijia Geng , Yuekang Li , Yunlan Du , Jun Xu , Yang Liu , and Bing Mao . 2020. An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March 2020 ). https:\/\/arxiv.org\/abs\/2003.09561v1 Sijia Geng, Yuekang Li, Yunlan Du, Jun Xu, Yang Liu, and Bing Mao. 2020. An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March 2020). https:\/\/arxiv.org\/abs\/2003.09561v1"},{"key":"e_1_3_2_1_25_1","unstructured":"Google. 2017. Fuzzer Test Suite. Google. https:\/\/github.com\/google\/fuzzer-test-suite  Google. 2017. Fuzzer Test Suite. Google. https:\/\/github.com\/google\/fuzzer-test-suite"},{"key":"e_1_3_2_1_26_1","unstructured":"Google. 2021. Automate your workflow from idea to production. https:\/\/github.com\/google\/fuzzing\/actions  Google. 2021. Automate your workflow from idea to production. https:\/\/github.com\/google\/fuzzing\/actions"},{"key":"e_1_3_2_1_27_1","unstructured":"Google. 2021. Coverage Guided Fuzz Testing. https:\/\/docs.gitlab.com\/ee\/user\/application_security\/coverage_fuzzing\/  Google. 2021. Coverage Guided Fuzz Testing. https:\/\/docs.gitlab.com\/ee\/user\/application_security\/coverage_fuzzing\/"},{"key":"e_1_3_2_1_28_1","unstructured":"Google. 2021. fuzzing - Google Scholar. https:\/\/scholar.google.com\/scholar?q=fuzzing&hl=en&as_sdt=0  Google. 2021. fuzzing - Google Scholar. https:\/\/scholar.google.com\/scholar?q=fuzzing&hl=en&as_sdt=0"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3427228.3427266"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428334"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_3_2_1_32_1","volume-title":"UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers. arXiv:2010.01785 [cs] (Oct.","author":"Li Yuwei","year":"2020","unstructured":"Yuwei Li , Shouling Ji , Yuan Chen , Sizhuang Liang , Wei-Han Lee , Yueyao Chen , Chenyang Lyu , Chunming Wu , Raheem Beyah , Peng Cheng , Kangjie Lu , and Ting Wang . 2020 . UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers. arXiv:2010.01785 [cs] (Oct. 2020). arxiv:2010.01785 [cs] http:\/\/arxiv.org\/abs\/2010.01785 Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, Peng Cheng, Kangjie Lu, and Ting Wang. 2020. UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers. arXiv:2010.01785 [cs] (Oct. 2020). arxiv:2010.01785 [cs] http:\/\/arxiv.org\/abs\/2010.01785"},{"key":"e_1_3_2_1_33_1","volume-title":"28th $USENIX$ Security Symposium ($USENIX$ Security 19). 1949\u20131966.","author":"Lyu Chenyang","unstructured":"Chenyang Lyu , Shouling Ji , Chao Zhang , Yuwei Li , Wei-Han Lee , Yu Song , and Raheem Beyah . 2019. $MOPT$ : Optimized mutation scheduling for fuzzers . In 28th $USENIX$ Security Symposium ($USENIX$ Security 19). 1949\u20131966. Chenyang Lyu, Shouling Ji, Chao Zhang, Yuwei Li, Wei-Han Lee, Yu Song, and Raheem Beyah. 2019. $MOPT$: Optimized mutation scheduling for fuzzers. In 28th $USENIX$ Security Symposium ($USENIX$ Security 19). 1949\u20131966."},{"key":"e_1_3_2_1_34_1","unstructured":"Jonathan Metzman and Asra Ali. 2021. Developers are Buzzing on Fuzzing. https:\/\/thenewstack.io\/developers-are-buzzing-on-fuzzing\/  Jonathan Metzman and Asra Ali. 2021. Developers are Buzzing on Fuzzing. https:\/\/thenewstack.io\/developers-are-buzzing-on-fuzzing\/"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Jonathan Metzman Abhishek Arya and L\u00e1szl\u00f3 Szekeres. 2020. FuzzBench: Fuzzer Benchmarking as a Service. https:\/\/security.googleblog.com\/2020\/03\/fuzzbench-fuzzer-benchmarking-as-service.html  Jonathan Metzman Abhishek Arya and L\u00e1szl\u00f3 Szekeres. 2020. FuzzBench: Fuzzer Benchmarking as a Service. https:\/\/security.googleblog.com\/2020\/03\/fuzzbench-fuzzer-benchmarking-as-service.html","DOI":"10.1145\/3468264.3473932"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2021.24118"},{"key":"e_1_3_2_1_37_1","unstructured":"Kostya Serebryany. 2015. libFuzzer a Library for Coverage-Guided Fuzz Testing. https:\/\/llvm.org\/docs\/LibFuzzer.html  Kostya Serebryany. 2015. libFuzzer a Library for Coverage-Guided Fuzz Testing. https:\/\/llvm.org\/docs\/LibFuzzer.html"},{"key":"e_1_3_2_1_38_1","unstructured":"Kostya Serebryany. 2021. Fuzzer Test Suite. https:\/\/github.com\/google\/fuzzer-test-suite.  Kostya Serebryany. 2021. Fuzzer Test Suite. https:\/\/github.com\/google\/fuzzer-test-suite."},{"key":"e_1_3_2_1_39_1","volume-title":"Proceedings of the 2012 USENIX Conference on Annual Technical Conference (USENIX ATC\u201912)","author":"Serebryany Konstantin","year":"2012","unstructured":"Konstantin Serebryany , Derek Bruening , Alexander Potapenko , and Dmitry Vyukov . 2012 . AddressSanitizer: A Fast Address Sanity Checker . In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (USENIX ATC\u201912) . USENIX Association, USA, 28. Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (USENIX ATC\u201912). USENIX Association, USA, 28."},{"key":"e_1_3_2_1_40_1","unstructured":"Paras Shah. 2021. Continuous Fuzzing with Go at Dgraph. https:\/\/dgraph.io\/blog\/post\/continuous-fuzzing-with-go\/  Paras Shah. 2021. Continuous Fuzzing with Go at Dgraph. https:\/\/dgraph.io\/blog\/post\/continuous-fuzzing-with-go\/"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP48549.2020.00011"},{"key":"e_1_3_2_1_42_1","unstructured":"Adith Sudhakar Mohit Arora and Souheil Moghnie. 2021. Focus on Fuzzing: Fuzzing Within the SDLC. https:\/\/safecode.org\/focus-on-fuzzing-fuzzing-within-the-sdlc\/  Adith Sudhakar Mohit Arora and Souheil Moghnie. 2021. Focus on Fuzzing: Fuzzing Within the SDLC. https:\/\/safecode.org\/focus-on-fuzzing-fuzzing-within-the-sdlc\/"},{"key":"e_1_3_2_1_43_1","unstructured":"Robert Swiecki. 2015. Honggfuzz. https:\/\/honggfuzz.dev\/  Robert Swiecki. 2015. Honggfuzz. https:\/\/honggfuzz.dev\/"},{"key":"e_1_3_2_1_44_1","unstructured":"Robert Swiecki. 2020. Improving honggfuzz for fuzzbench part 3. https:\/\/groups.google.com\/g\/fuzzing-discuss\/c\/rv59P0svXjI  Robert Swiecki. 2020. Improving honggfuzz for fuzzbench part 3. https:\/\/groups.google.com\/g\/fuzzing-discuss\/c\/rv59P0svXjI"},{"key":"e_1_3_2_1_45_1","unstructured":"Micha\u0142 Zalewski. 2014. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/  Micha\u0142 Zalewski. 2014. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_3_2_1_46_1","unstructured":"Micha\u0142 Zalewski. 2014. Fuzzing with Afl-Fuzz AFL 2.53b Documentation. https:\/\/afl-1.readthedocs.io\/en\/latest\/fuzzing.html  Micha\u0142 Zalewski. 2014. Fuzzing with Afl-Fuzz AFL 2.53b Documentation. https:\/\/afl-1.readthedocs.io\/en\/latest\/fuzzing.html"},{"key":"e_1_3_2_1_47_1","unstructured":"Andreas Zeller. 2019. Andreas Zeller\u2019s Old Blog: When Results Are All That Matters: Consequences. https:\/\/andreas-zeller.blogspot.com\/2019\/10\/when-results-are-all-that-matters.html  Andreas Zeller. 2019. Andreas Zeller\u2019s Old Blog: When Results Are All That Matters: Consequences. https:\/\/andreas-zeller.blogspot.com\/2019\/10\/when-results-are-all-that-matters.html"},{"key":"e_1_3_2_1_48_1","unstructured":"Andreas Zeller. 2019. Andreas Zeller\u2019s Old Blog: When Results Are All That Matters: The Case of the Angora Fuzzer. https:\/\/andreas-zeller.blogspot.com\/2019\/10\/when-results-are-all-that-matters-case.html  Andreas Zeller. 2019. Andreas Zeller\u2019s Old Blog: When Results Are All That Matters: The Case of the Angora Fuzzer. https:\/\/andreas-zeller.blogspot.com\/2019\/10\/when-results-are-all-that-matters-case.html"}],"event":{"name":"ESEC\/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Athens Greece","acronym":"ESEC\/FSE '21","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3473932","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3468264.3473932","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:23Z","timestamp":1750191443000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3473932"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,18]]},"references-count":48,"alternative-id":["10.1145\/3468264.3473932","10.1145\/3468264"],"URL":"https:\/\/doi.org\/10.1145\/3468264.3473932","relation":{},"subject":[],"published":{"date-parts":[[2021,8,18]]},"assertion":[{"value":"2021-08-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}