{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T21:08:56Z","timestamp":1782940136769,"version":"3.54.5"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,9,30]],"date-time":"2023-09-30T00:00:00Z","timestamp":1696032000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Research Council (ERC) under the European Union\u2019s Horizon 2020 research and innovation programme","award":["864972"],"award-info":[{"award-number":["864972"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,11,30]]},"abstract":"<jats:p>RESTful APIs are a type of web service that are widely used in industry. In the past few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real faults were automatically found in a large variety of RESTful APIs. However, usually the analyzed fuzzers treat the APIs as black-box, and no analysis of what is actually covered in these systems is done. Therefore, although these fuzzers are clearly useful for practitioners, we do not know their current limitations and actual effectiveness. Solving this is a necessary step to be able to design better, more efficient, and effective techniques. To address this issue, in this article we compare seven state-of-the-art fuzzers on 18 open source\u20141 industrial and 1 artificial\u2014RESTful APIs. We then analyze the source code for which parts of these APIs the fuzzers fail to generate tests. This analysis points to clear limitations of these current fuzzers, listing concrete follow-up challenges for the research community.<\/jats:p>","DOI":"10.1145\/3597205","type":"journal-article","created":{"date-parts":[[2023,5,13]],"date-time":"2023-05-13T11:15:54Z","timestamp":1683976554000},"page":"1-45","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["Open Problems in Fuzzing RESTful APIs: A Comparison of Tools"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1204-9322","authenticated-orcid":false,"given":"Man","family":"Zhang","sequence":"first","affiliation":[{"name":"Kristiania University College, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0799-2930","authenticated-orcid":false,"given":"Andrea","family":"Arcuri","sequence":"additional","affiliation":[{"name":"Kristiania University College and Oslo Metropolitan University, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,30]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Amazon Gateway API. n.d. Working with REST APIs. Retrieved May 20 2022 from https:\/\/docs.aws.amazon.com\/apigateway\/latest\/developerguide\/apigateway-rest-api.html."},{"key":"e_1_3_3_3_2","unstructured":"GitHub. n.d. APIFuzzer\u2014HTTP API Testing Framework. Retrieved May 23 2023 from https:\/\/github.com\/KissPeter\/APIFuzzer."},{"key":"e_1_3_3_4_2","unstructured":"GitHub. n.d. ApiTester. Retrieved May 20 2022 from https:\/\/github.com\/opendata-for-all\/api-tester."},{"key":"e_1_3_3_5_2","unstructured":"Nuno Laranjeiro. n.d. bBOXRT. Retrieved May 23 2023 from https:\/\/git.dei.uc.pt\/cnl\/bBOXRT."},{"key":"e_1_3_3_6_2","unstructured":"GitHub. n.d. C8. Retrieved May 20 2022 from https:\/\/github.com\/bcoe\/c8."},{"key":"e_1_3_3_7_2","unstructured":"GitHub. n.d. EvoMaster. Retrieved May 23 2023 from https:\/\/github.com\/EMResearch\/EvoMaster."},{"key":"e_1_3_3_8_2","unstructured":"GitHub. n.d. EvoMaster Benchmark (EMB). Retrieved May 20 2022 from https:\/\/github.com\/EMResearch\/EMB."},{"key":"e_1_3_3_9_2","unstructured":"Google Drive. n.d. Introduction to Google Drive API. Retrieved May 20 2022 from https:\/\/developers.google.com\/drive\/api\/v3\/about-sdk."},{"key":"e_1_3_3_10_2","unstructured":"Hibernate. n.d. Hibernate Home Page. Retrieved May 20 2022 from http:\/\/hibernate.org."},{"key":"e_1_3_3_11_2","unstructured":"JaCoCo. n.d. JaCoCo Home Page. Retrieved May 20 2022 from https:\/\/www.jacoco.org\/."},{"key":"e_1_3_3_12_2","unstructured":"GitHub. n.d. Language-Agnostic HTTP API Testing Tool. Retrieved May 23 2023 from https:\/\/github.com\/apiaryio\/dredd."},{"key":"e_1_3_3_13_2","unstructured":"Microsoft. n.d. LinkedIn API. Retrieved May 20 2022 from https:\/\/docs.microsoft.com\/en-us\/linkedin\/."},{"key":"e_1_3_3_14_2","unstructured":"Swagger. n.d. OpenAPI\/Swagger. Retrieved May 23 2023 from https:\/\/swagger.io\/."},{"key":"e_1_3_3_15_2","unstructured":"Reddit. n.d. Reddit API. Retrieved May 20 2022 from https:\/\/www.reddit.com\/dev\/api."},{"key":"e_1_3_3_16_2","unstructured":"GitHub. n.d. RestCT. Retrieved May 23 2023 from https:\/\/github.com\/GIST-NJU\/RestCT."},{"key":"e_1_3_3_17_2","unstructured":"GitHub. n.d. RESTest. Retrieved May 23 2023 from https:\/\/github.com\/isa-group\/RESTest."},{"key":"e_1_3_3_18_2","unstructured":"GitHub. n.d. RESTler. Retrieved May 23 2023 from https:\/\/github.com\/microsoft\/restler-fuzzer."},{"key":"e_1_3_3_19_2","unstructured":"GitHub. n.d. RestTestGen. Retrieved May 23 2023 from https:\/\/github.com\/SeUniVr\/RestTestGen."},{"key":"e_1_3_3_20_2","unstructured":"Schemathesi. n.d. Command Line Interface. Retrieved May 23 2023 from https:\/\/schemathesis.readthedocs.io\/en\/stable\/cli.html."},{"key":"e_1_3_3_21_2","unstructured":"Schemathesis. n.d. Schemathesis: Property-Based Testing for API Schemas. Retrieved May 23 2023 from https:\/\/schemathesis.readthedocs.io\/."},{"key":"e_1_3_3_22_2","unstructured":"GitHub. n.d. Tcases for OpenAPI: From REST-ful to Test-ful. Retrieved May 23 2023 from https:\/\/github.com\/Cornutum\/tcases\/tree\/master\/tcases-openapi."},{"key":"e_1_3_3_23_2","unstructured":"Twitter Developer Platform. n.d. Twitter API. Retrieved May 20 2022 from https:\/\/developer.twitter.com\/en\/docs\/twitter-api."},{"key":"e_1_3_3_24_2","unstructured":"WireMock. n.d. WireMock Home Page. Retrieved May 20 2022 from https:\/\/wiremock.org\/."},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3390194"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.16"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1569901.1570122"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2017.11"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00046"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3013820"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3391533"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477271"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02153"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6651631"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6106830"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00083"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00046"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520304.3528952"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2209.05833"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510230"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.010"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM52516.2021.00035"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1808"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDOC.2018.00031"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409719"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397374"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2212.14604"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.71"},{"key":"e_1_3_3_51_2","first-page":"345","volume-title":"Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion\u201922)","author":"Hatfield-Dodds Zac","year":"2022","unstructured":"Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving semantics-aware fuzzers from web API schemas. In Proceedings of the 2022 IEEE\/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion\u201922). IEEE, Los Alamitos, CA, 345\u2013346."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00023"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2204.08348"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3056505"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510133"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01891"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491038"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00034"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2021.3050610"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65310-1_33"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3469082"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/DeepTest52559.2021.00008"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2018.2847344"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.5555\/2904388"},{"key":"e_1_3_3_66_2","article-title":"ARTE: Automated generation of realistic test inputs for web APIs","author":"Valenzuela Juan Carlos Alonso","year":"2023","unstructured":"Juan Carlos Alonso Valenzuela, Alberto Martin-Lopez, Sergio Segura, Jose Maria Garcia, and Antonio Ruiz-Cortes. 2023. ARTE: Automated generation of realistic test inputs for web APIs. IEEE Transactions on Software Engineering 49, 1 (2023), 348\u2013363.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00024"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510151"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693084"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985876"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464940"},{"key":"e_1_3_3_72_2","article-title":"Open problems in fuzzing RESTful APIs: A comparison of tools","author":"Zhang Man","year":"2022","unstructured":"Man Zhang and Andrea Arcuri. 2022. Open problems in fuzzing RESTful APIs: A comparison of tools. arXiv preprint arXiv:2205.05325 (2022).","journal-title":"arXiv preprint arXiv:2205.05325"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2208.12743"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3585009"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00022"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09937-1"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597205","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597205","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:06Z","timestamp":1750182546000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597205"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,30]]},"references-count":75,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11,30]]}},"alternative-id":["10.1145\/3597205"],"URL":"https:\/\/doi.org\/10.1145\/3597205","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,30]]},"assertion":[{"value":"2022-07-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}