{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:45:03Z","timestamp":1787017503359,"version":"build-2736575974"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,11,23]],"date-time":"2023-11-23T00:00:00Z","timestamp":1700697600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"JSPS and SNSF","award":["183587, JPJSJRP20191502"],"award-info":[{"award-number":["183587, JPJSJRP20191502"]}]},{"name":"JSPS","award":["JP21H04877, JP22K18630, JP21K17725"],"award-info":[{"award-number":["JP21H04877, JP22K18630, JP21K17725"]}]},{"name":"JST","award":["JPMJPR22P3"],"award-info":[{"award-number":["JPMJPR22P3"]}]},{"name":"Kyushu University for the Leading Human Resource Development Fellowship"},{"name":"Inamori Research Institute for Science"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>Fuzz testing, also known as fuzzing, is a software testing technique aimed at identifying software vulnerabilities. In recent decades, fuzzing has gained increasing popularity in the research community. However, existing studies led by fuzzing experts mainly focus on improving the coverage and performance of fuzzing techniques. That is, there is still a gap in empirical knowledge regarding fuzzing, especially about the challenges developers face when they adopt fuzzing. Understanding these challenges can provide valuable insights to both practitioners and researchers on how to further improve fuzzing processes and techniques.<\/jats:p>\n                  <jats:p>We conducted a study to understand the challenges encountered by developers during fuzzing. More specifically, we first manually analyzed 829 randomly sampled fuzzing-related GitHub issues and constructed a taxonomy consisting of 39 types of challenges (22 related to the fuzzing process itself, 17 related to using external fuzzing providers). We then surveyed 106 fuzzing practitioners to verify the validity of our taxonomy and collected feedback on how the fuzzing process can be improved. Our taxonomy, accompanied with representative examples and highlighted implications, can serve as a reference point on how to better adopt fuzzing techniques for practitioners, and indicates potential directions researchers can work on toward better fuzzing approaches and practices.<\/jats:p>\n                  <jats:p\/>","DOI":"10.1145\/3611668","type":"journal-article","created":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T08:31:48Z","timestamp":1690965108000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["The Human Side of Fuzzing: Challenges Faced by Developers during Fuzzing Activities"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7100-9227","authenticated-orcid":false,"given":"Olivier","family":"Nourry","sequence":"first","affiliation":[{"name":"Kyushu University, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9633-7577","authenticated-orcid":false,"given":"Yutaro","family":"Kashiwa","sequence":"additional","affiliation":[{"name":"Nara Institute of Science and Technolog, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6307-8460","authenticated-orcid":false,"given":"Bin","family":"Lin","sequence":"additional","affiliation":[{"name":"Radboud University, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2216-3148","authenticated-orcid":false,"given":"Gabriele","family":"Bavota","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4391-0197","authenticated-orcid":false,"given":"Michele","family":"Lanza","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7058-1045","authenticated-orcid":false,"given":"Yasutaka","family":"Kamei","sequence":"additional","affiliation":[{"name":"Kyushu University, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,11,23]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Anaconda. 2022. State of Data Science Report 2022. https:\/\/www.anaconda.com\/state-of-data-science-report-2022"},{"key":"e_1_3_2_3_2","article-title":"Apache software flaw could result in major breaches","author":"Analytica Oxford","year":"2021","unstructured":"Oxford Analytica. 2021. Apache software flaw could result in major breaches. Emerald Expert Briefingsoxan-es (2021).","journal-title":"Emerald Expert Briefings"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP40000.2020.00117"},{"key":"e_1_3_2_5_2","unstructured":"bitcoin\/bitcoin. 2020. Issue #19557. https:\/\/github.com\/bitcoin\/bitcoin\/issues\/19557. Last accessed on 07-12-2022."},{"key":"e_1_3_2_6_2","unstructured":"bitcoin\/bitcoin. 2020. Issue #20088. https:\/\/github.com\/bitcoin\/bitcoin\/issues\/20088. Last accessed on 07-12-2022."},{"key":"e_1_3_2_7_2","unstructured":"bitcoin\/bitcoin. 2020. Issue #20232. https:\/\/github.com\/bitcoin\/bitcoin\/issues\/20232. Last accessed on 07-12-2022."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3016773"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409729"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134020"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978428"},{"key":"e_1_3_2_12_2","unstructured":"bytecodealliance\/wasmtime. 2019. Issue #459. https:\/\/github.com\/bytecodealliance\/wasmtime\/issues\/459. Last accessed on 07-12-2022."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2014.66"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2015.50"},{"issue":"4","key":"e_1_3_2_15_2","first-page":"392","article-title":"Questionnaire design, interviewing and attitude measurement","volume":"35","author":"Chisnall Peter M.","year":"1993","unstructured":"Peter M. Chisnall. 1993. Questionnaire design, interviewing and attitude measurement. Journal of the Market Research Society 35, 4 (1993), 392\u2013393.","journal-title":"Journal of the Market Research Society"},{"key":"e_1_3_2_16_2","unstructured":"Chronium. [n. d.]. Monorail Chronium. https:\/\/bugs.chromium.org\/. Last accessed on 07-12-2022."},{"key":"e_1_3_2_17_2","unstructured":"ClickHouse\/ClickHouse. 2021. Issue #32443. https:\/\/github.com\/ClickHouse\/ClickHouse\/issues\/32443. Last accessed on 07-12-2022."},{"key":"e_1_3_2_18_2","volume-title":"Biostatistics: A Foundation for Analysis in the Health Sciences","author":"Daniel Wayne W.","year":"2018","unstructured":"Wayne W. Daniel and Chad L. Cross. 2018. Biostatistics: A Foundation for Analysis in the Health Sciences. Wiley."},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Zhen Yu Ding and Claire Le Goues. 2021. An Empirical Study of OSS-Fuzz Bugs. https:\/\/arxiv.org\/abs\/2103.11518","DOI":"10.1109\/MSR52588.2021.00026"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510197"},{"key":"e_1_3_2_21_2","unstructured":"ethereum\/solidity. 2019. Issue #6201. https:\/\/github.com\/ethereum\/solidity\/issues\/6201. Last accessed on 07-12-2022."},{"key":"e_1_3_2_22_2","unstructured":"ethereum\/solidity. 2019. Issue #7948. https:\/\/github.com\/ethereum\/solidity\/issues\/7948. Last accessed on 07-12-2022."},{"key":"e_1_3_2_23_2","unstructured":"ethereum\/solidity. 2021. Issue #6587. https:\/\/github.com\/ethereum\/solidity\/issues\/6587. Last accessed on 07-12-2022."},{"key":"e_1_3_2_24_2","unstructured":"golang\/go. 2021. Issue #48179. https:\/\/github.com\/golang\/go\/issues\/48179. Last accessed on 07-12-2022."},{"key":"e_1_3_2_25_2","unstructured":"golang\/go. 2022. Issue #52292. https:\/\/github.com\/golang\/go\/issues\/52292. Last accessed on 07-12-2022."},{"key":"e_1_3_2_26_2","unstructured":"Google. [n. d.]. Geting Started Bug disclosure guidelines OSS-Fuzz. https:\/\/google.github.io\/oss-fuzz\/getting-started\/bug-disclosure-guidelines\/. Last accessed on 07-12-2022."},{"key":"e_1_3_2_27_2","unstructured":"Google. [n. d.]. Geting Started OSS-Fuzz. https:\/\/google.github.io\/oss-fuzz\/getting-started\/. Last accessed on 07-12-2022."},{"key":"e_1_3_2_28_2","unstructured":"Google. [n. d.]. OSS-Fuzz. https:\/\/google.github.io\/oss-fuzz\/"},{"key":"e_1_3_2_29_2","unstructured":"google\/oss fuzz. [n. d.]. Issue #5526. https:\/\/github.com\/google\/oss-fuzz\/issues\/5526. Last accessed on 07-12-2022."},{"key":"e_1_3_2_30_2","unstructured":"google\/oss fuzz. 2016. Issue #153. https:\/\/github.com\/google\/oss-fuzz\/issues\/153. Last accessed on 07-12-2022."},{"key":"e_1_3_2_31_2","unstructured":"google\/oss fuzz. 2017. Issue #243. https:\/\/github.com\/google\/oss-fuzz\/issues\/243. Last accessed on 07-12-2022."},{"key":"e_1_3_2_32_2","unstructured":"google\/oss fuzz. 2017. Issue #244. https:\/\/github.com\/google\/oss-fuzz\/issues\/244. Last accessed on 07-12-2022."},{"key":"e_1_3_2_33_2","unstructured":"google\/oss fuzz. 2017. Issue #273. https:\/\/github.com\/google\/oss-fuzz\/issues\/273. Last accessed on 07-12-2022."},{"key":"e_1_3_2_34_2","unstructured":"google\/oss fuzz. 2017. Issue #638. https:\/\/github.com\/google\/oss-fuzz\/issues\/638. Last accessed on 07-12-2022."},{"key":"e_1_3_2_35_2","unstructured":"google\/oss fuzz. 2017. Issue #815. https:\/\/github.com\/google\/oss-fuzz\/issues\/815. Last accessed on 07-12-2022."},{"key":"e_1_3_2_36_2","unstructured":"google\/oss fuzz. 2017. Issue #823. https:\/\/github.com\/google\/oss-fuzz\/issues\/823. Last accessed on 07-12-2022."},{"key":"e_1_3_2_37_2","unstructured":"google\/oss fuzz. 2018. Issue #1333. https:\/\/github.com\/google\/oss-fuzz\/issues\/1333. Last accessed on 07-12-2022."},{"key":"e_1_3_2_38_2","unstructured":"google\/oss fuzz. 2018. Issue #1344. https:\/\/github.com\/google\/oss-fuzz\/issues\/1344. Last accessed on 07-12-2022."},{"key":"e_1_3_2_39_2","unstructured":"google\/oss fuzz. 2018. Issue #1675. https:\/\/github.com\/google\/oss-fuzz\/issues\/1675. Last accessed on 07-12-2022."},{"key":"e_1_3_2_40_2","unstructured":"google\/oss fuzz. 2019. Issue #2094. https:\/\/github.com\/google\/oss-fuzz\/issues\/2094. Last accessed on 07-12-2022."},{"key":"e_1_3_2_41_2","unstructured":"google\/oss fuzz. 2019. Issue #2558. https:\/\/github.com\/google\/oss-fuzz\/issues\/2558. Last accessed on 07-12-2022."},{"key":"e_1_3_2_42_2","unstructured":"google\/oss fuzz. 2019. Issue #2635. https:\/\/github.com\/google\/oss-fuzz\/issues\/2635. Last accessed on 07-12-2022."},{"key":"e_1_3_2_43_2","unstructured":"google\/oss fuzz. 2019. Issue #3014. https:\/\/github.com\/google\/oss-fuzz\/issues\/3014. Last accessed on 07-12-2022."},{"key":"e_1_3_2_44_2","unstructured":"google\/oss fuzz. 2019. Issue #3082. https:\/\/github.com\/google\/oss-fuzz\/issues\/3082. Last accessed on 07-12-2022."},{"key":"e_1_3_2_45_2","unstructured":"google\/oss fuzz. 2020. Issue #3227. https:\/\/github.com\/google\/oss-fuzz\/issues\/3227. Last accessed on 07-12-2022."},{"key":"e_1_3_2_46_2","unstructured":"google\/oss fuzz. 2020. Issue #3548. https:\/\/github.com\/google\/oss-fuzz\/issues\/3548. Last accessed on 07-12-2022."},{"key":"e_1_3_2_47_2","unstructured":"google\/oss fuzz. 2021. Issue #3432. https:\/\/github.com\/google\/oss-fuzz\/issues\/3432. Last accessed on 07-12-2022."},{"key":"e_1_3_2_48_2","unstructured":"google\/oss fuzz. 2021. Issue #6147. https:\/\/github.com\/google\/oss-fuzz\/issues\/6147. Last accessed on 07-12-2022."},{"key":"e_1_3_2_49_2","unstructured":"google\/oss fuzz. 2021. Issue #6871. https:\/\/github.com\/google\/oss-fuzz\/issues\/6871. Last accessed on 07-12-2022."},{"key":"e_1_3_2_50_2","unstructured":"google\/oss fuzz. 2021. Issue #6885. https:\/\/github.com\/google\/oss-fuzz\/issues\/6885. Last accessed on 07-12-2022."},{"key":"e_1_3_2_51_2","unstructured":"google\/oss fuzz. 2022. Issue #7138. https:\/\/github.com\/google\/oss-fuzz\/issues\/7138. Last accessed on 07-12-2022."},{"key":"e_1_3_2_52_2","unstructured":"google\/oss fuzz. 2022. Issue #7198. https:\/\/github.com\/google\/oss-fuzz\/issues\/7198. Last accessed on 07-12-2022."},{"key":"e_1_3_2_53_2","unstructured":"google\/oss fuzz. 2022. Issue #7581. https:\/\/github.com\/google\/oss-fuzz\/issues\/7581. Last accessed on 07-12-2022."},{"key":"e_1_3_2_54_2","unstructured":"google\/syzkaller. 2016. Issue #57. https:\/\/github.com\/google\/syzkaller\/issues\/57. Last accessed on 07-12-2022."},{"key":"e_1_3_2_55_2","unstructured":"google\/syzkaller. 2020. Issue #1622. https:\/\/github.com\/google\/syzkaller\/issues\/1622. Last accessed on 07-12-2022."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510228"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510092"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.23919\/MIPRO.2018.8400261"},{"key":"e_1_3_2_59_2","unstructured":"ISO\/IEC. [n. d.]. ISO 25000 Portal. https:\/\/iso25000.com\/index.php\/en\/iso-25000-standards\/iso-25010"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2019.8870150"},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","unstructured":"Thijs Klooster Fatih Turkmen Gerben Broenink Ruben ten Hove and Marcel B\u00f6hme. 2022. Effectiveness and Scalability of Fuzzing Techniques in CI\/CD Pipelines. https:\/\/arxiv.org\/abs\/2205.14964","DOI":"10.1109\/SBFT59156.2023.00015"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-005-1290-x"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510208"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2018.2834476"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330260"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3100858"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510082"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510182"},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","unstructured":"Olivier Nourry Yutaro Kashiwa Bin Lin Gabriele Bavota Michele Lanza and Kamei Yasutaka. 2023. Replication package for the paper: The Human Side of Fuzzing: Challenges Faced by Developers During Fuzzing Activities. https:\/\/github.com\/posl\/fuzzingChallengesReplicationPackage","DOI":"10.1145\/3611668"},{"key":"e_1_3_2_70_2","unstructured":"obgm\/libcoap. 2018. Issue #213. https:\/\/github.com\/obgm\/libcoap\/issues\/213. Last accessed on 07-12-2022."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360600"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00056"},{"key":"e_1_3_2_73_2","first-page":"861","volume-title":"Proceedings of the 23rd USENIX Security Symposium (USENIX Security\u201914)","author":"Rebert Alexandre","year":"2014","unstructured":"Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, and David Brumley. 2014. Optimizing seed selection for fuzzing. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security\u201914). 861\u2013875."},{"key":"e_1_3_2_74_2","unstructured":"secdev\/scapy. 2018. Issue #1398. https:\/\/github.com\/secdev\/scapy\/issues\/1398. Last accessed on 07-12-2022."},{"key":"e_1_3_2_75_2","unstructured":"secdev\/scapy. 2021. Issue #3347. https:\/\/github.com\/secdev\/scapy\/issues\/3347. Last accessed on 07-12-2022."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134105"},{"key":"e_1_3_2_77_2","volume-title":"Card Sorting: Designing Usable Categories","author":"Spencer Donna","year":"2009","unstructured":"Donna Spencer. 2009. Card Sorting: Designing Usable Categories. Rosenfeld Media."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSC55868.2022.00058"},{"key":"e_1_3_2_80_2","doi-asserted-by":"crossref","unstructured":"Joobeom Yun Fayozbek Rustamov Juhwan Kim and Youngjoo Shin. 2022. Fuzzing of embedded systems: A survey. 55 7 (2022).","DOI":"10.1145\/3538644"},{"key":"e_1_3_2_81_2","unstructured":"Michael Zalewski. [n. d.]. American fuzzy lop (2.52b). https:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3124006"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00106"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460120.3484596"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611668","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3611668","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T12:37:08Z","timestamp":1750163828000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611668"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,23]]},"references-count":83,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3611668"],"URL":"https:\/\/doi.org\/10.1145\/3611668","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,23]]},"assertion":[{"value":"2023-02-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}