{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T23:21:09Z","timestamp":1769728869661,"version":"3.49.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62472209"],"award-info":[{"award-number":["62472209"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"publisher","award":["BK20221439"],"award-info":[{"award-number":["BK20221439"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100013058","name":"Primary Research and Development Plan of Jiangsu Province","doi-asserted-by":"crossref","award":["BE2023025-2"],"award-info":[{"award-number":["BE2023025-2"]}],"id":[{"id":"10.13039\/501100013058","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>REST APIs are essential for building modern enterprise systems, but effectively testing them remains challenging, particularly due to difficulties in inferring constraints from specifications. Current testing approaches typically use feedback from HTTP status codes to guide input generation. However, they overlook valuable information available in the accompanying error messages, reducing their effectiveness in exploring the APIs\u2019 input spaces. In this paper, we propose EmRest, a black-box testing approach that leverages error message analysis to enhance both valid and exceptional test input generation for REST APIs. For each operation under test, EmRest first identifies all possible value assignment strategies for each of its input parameters. It then repeatedly applies combinatorial testing to sample test inputs based on these strategies, and statistically analyzes the error messages (of 400-range status code) received to infer and exclude invalid combinations of value assignment strategies (i.e., constraints of the input space). Additionally, EmRest seeks to mutate valid value assignment strategies that are finally identified to generate test inputs for exceptional testing. The error messages (of 500-range status code) received are categorized to identify bug-prone operations, for which more testing resources are allocated. Our experimental results on 16 real-world REST APIs demonstrates the effectiveness of EmRest. It achieves higher operation coverage than state-of-the-art approaches in 50% of APIs, and detects 226 unique bugs undetected by other approaches.<\/jats:p>","DOI":"10.1145\/3728964","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1978-2000","source":"Crossref","is-referenced-by-count":1,"title":["Effective REST APIs Testing with Error Message Analysis"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-6085-345X","authenticated-orcid":false,"given":"Lixin","family":"Xu","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1383-5421","authenticated-orcid":false,"given":"Huayao","family":"Wu","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-4385-0908","authenticated-orcid":false,"given":"Zhenyu","family":"Pan","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4323-497X","authenticated-orcid":false,"given":"Tongtong","family":"Xu","sequence":"additional","affiliation":[{"name":"Huawei, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5777-7759","authenticated-orcid":false,"given":"Shaohua","family":"Wang","sequence":"additional","affiliation":[{"name":"Central University of Finance and Economics, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5786-0894","authenticated-orcid":false,"given":"Xintao","family":"Niu","sequence":"additional","affiliation":[{"name":"Nanjing University, Suzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9575-1012","authenticated-orcid":false,"given":"Changhai","family":"Nie","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. Built-in String Formats of OpenAPI. https:\/\/swagger.io\/docs\/specification\/data-models\/data-types\/##string Accessed: 2024-07-02"},{"key":"e_1_2_1_2_1","unstructured":"2024. Dredd. https:\/\/github.com\/apiaryio\/dredd Accessed: 2024-10-30"},{"key":"e_1_2_1_3_1","unstructured":"2024. FuzzyWuzzy. https:\/\/github.com\/seatgeek\/fuzzywuzzy Accessed: 2024-07-04"},{"key":"e_1_2_1_4_1","unstructured":"2024. JaCoCo. https:\/\/www.eclemma.org\/jacoco\/ Accessed: 2024-07-02"},{"key":"e_1_2_1_5_1","unstructured":"2024. JSON. https:\/\/www.json.org\/json-en.html Accessed: 2024-07-04"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"2024. MITMProxy. https:\/\/mitmproxy.org\/ Accessed: 2024-07-04","DOI":"10.53414\/UIJES.2024.4.4.17"},{"key":"e_1_2_1_7_1","unstructured":"2024. The Official YAML Web Site. https:\/\/yaml.org\/ Accessed: 2024-07-04"},{"key":"e_1_2_1_8_1","unstructured":"2024. OpenAPI Specification - Version 3.1.0 | Swagger. https:\/\/swagger.io\/specification\/ Accessed: 2024-07-02"},{"key":"e_1_2_1_9_1","unstructured":"2024. PICT. https:\/\/github.com\/microsoft\/pict Accessed: 2024-07-04"},{"key":"e_1_2_1_10_1","unstructured":"2024. Tcases. https:\/\/github.com\/Cornutum\/tcases Accessed: 2024-10-30"},{"key":"e_1_2_1_11_1","unstructured":"2024. YouTube Data API Specification. https:\/\/api.apis.guru\/v2\/specs\/googleapis.com\/youtube\/v3\/openapi.yaml Accessed: 2024-10-29"},{"key":"e_1_2_1_12_1","unstructured":"2025. Replication package. https:\/\/zenodo.org\/records\/14940931 Accessed: 2025-02-28"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473491"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9570-9"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00083"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-48421-6_11"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCNS53852.2021.00023"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00213"},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","first-page":"e1808","DOI":"10.1002\/stvr.1808","article-title":"Automated black-box testing of nominal and error scenarios in RESTful APIs","volume":"32","author":"Corradini Davide","year":"2022","unstructured":"Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32, 5 (2022), e1808.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639106"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 21st USENIX Conference on Security Symposium (Security\u201912)","author":"Doup\u00e9 Adam","year":"2012","unstructured":"Adam Doup\u00e9, Ludovico Cavedon, Christopher Kruegel, and Giovanni Vigna. 2012. Enemy of the state: a state-aware black-box web vulnerability scanner. In Proceedings of the 21st USENIX Conference on Security Symposium (Security\u201912). USENIX Association, USA. 26."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/edoc.2018.00031"},{"key":"e_1_2_1_24_1","volume-title":"Taylor","author":"Fielding Roy Thomas","year":"2000","unstructured":"Roy Thomas Fielding and Richard N. Taylor. 2000. Architectural styles and the design of network-based software architectures. Ph. D. Dissertation. isbn:0599871180 AAI9980887"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409719"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3617175"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00016"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3528637"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3528637"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 32nd USENIX Conference on Security Symposium (SEC \u201923)","author":"Jiang Zu-Ming","year":"2023","unstructured":"Zu-Ming Jiang, Jia-Ju Bai, and Zhendong Su. 2023. DynSQL: Stateful fuzzing for database management systems with complex and valid SQL query generation. In Proceedings of the 32nd USENIX Conference on Security Symposium (SEC \u201923). USENIX Association, USA. Article 277, 17 pages. isbn:978-1-939133-37-3"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00023"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598131"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00218"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639769"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534401"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2004.24"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST60714.2024.00017"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-024-00745-2"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion52605.2021.00040"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00023"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510133"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 32nd USENIX Conference on Security Symposium (SEC \u201923)","author":"Lyu Chenyang","year":"2023","unstructured":"Chenyang Lyu, Jiacheng Xu, Shouling Ji, Xuhong Zhang, Qinying Wang, Binbin Zhao, Gaoning Pan, Wei Cao, Peng Chen, and Raheem Beyah. 2023. MINER: a hybrid data-driven approach for REST API fuzzing. In Proceedings of the 32nd USENIX Conference on Security Symposium (SEC \u201923). USENIX Association, USA. Article 253, 18 pages. isbn:978-1-939133-37-3"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2021.3050610"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-33702-5_31"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340433.3342822"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3469082"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/DeepTest52559.2021.00008"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1883612.1883618"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2865772"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","unstructured":"Mark Nottingham and Erik Wilde. 2016. Problem Details for HTTP APIs. RFC 7807. https:\/\/doi.org\/10.17487\/RFC7807 10.17487\/RFC7807","DOI":"10.17487\/RFC7807"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","unstructured":"Mark Nottingham Erik Wilde and Sanjay Dalal. 2023. Problem Details for HTTP APIs. RFC 9457. https:\/\/doi.org\/10.17487\/RFC9457 10.17487\/RFC9457","DOI":"10.17487\/RFC9457"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00062"},{"key":"e_1_2_1_53_1","unstructured":"Postman API Evangelist. 2024. Postman to OpenAPI Collection. https:\/\/www.postman.com\/api-evangelist\/artificial-intelligence\/collection\/txy0rdd\/postman-to-openapi Accessed: 2025-02-11"},{"key":"e_1_2_1_54_1","unstructured":"RobBagby. 2023. Web API Implementation - Best Practices for Cloud Applications. https:\/\/learn.microsoft.com\/en-us\/azure\/architecture\/best-practices\/api-implementation."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","unstructured":"Sergio Segura Jos\u00e9 A. Parejo Javier Troya and Antonio Ruiz-Cort\u00e9s. 2018. Metamorphic testing of RESTful web APIs. 882. isbn:9781450356381 https:\/\/doi.org\/10.1145\/3180155.3182528 10.1145\/3180155.3182528","DOI":"10.1145\/3180155.3182528"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695532"},{"key":"e_1_2_1_57_1","unstructured":"SpringDoc Contributors. 2024. SpringDoc OpenAPI. https:\/\/springdoc.org\/ Accessed: 2025-02-11"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678586"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00024"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00062"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510151"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597205"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728964","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:46:45Z","timestamp":1752684405000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728964"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":62,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728964"],"URL":"https:\/\/doi.org\/10.1145\/3728964","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}