{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T06:59:10Z","timestamp":1794034750608,"version":"4.8.0"},"reference-count":114,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>The number and complexity of test case generation tools for REST APIs have significantly increased in recent years. These tools excel in automating input generation but are limited by their test oracles, which can only detect crashes, regressions, and violations of API specifications or design best practices. This article introduces AGORA+, an approach for generating test oracles for REST APIs through the detection of invariants\u2014output properties that should always hold. AGORA+ learns the expected behavior of an API by analyzing API requests and their corresponding responses. We enhanced the Daikon tool for dynamic detection of likely invariants, adding new invariant types and creating a front-end called Beet. Beet translates any OpenAPI specification and a set of API requests and responses into Daikon inputs. AGORA+ can detect 106 different types of invariants in REST APIs. We also developed PostmanAssertify, which converts the invariants identified by AGORA+ into executable JavaScript assertions. AGORA+ achieved a precision of 80% on 25 operations from 20 industrial APIs. It also identified 48% of errors systematically seeded in the outputs of the APIs under test. AGORA+ uncovered 32 bugs in popular APIs, including Amadeus, Deutschebahn, GitHub, Marvel, NYTimesBooks, and YouTube, leading to fixes and documentation updates.<\/jats:p>","DOI":"10.1145\/3726524","type":"journal-article","created":{"date-parts":[[2025,3,28]],"date-time":"2025-03-28T14:46:34Z","timestamp":1743173194000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Test Oracle Generation for REST APIs"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1177-9262","authenticated-orcid":false,"given":"Juan C.","family":"Alonso","sequence":"first","affiliation":[{"name":"SCORE Lab, Universidad de Sevilla, Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9379-277X","authenticated-orcid":false,"given":"Michael D.","family":"Ernst","sequence":"additional","affiliation":[{"name":"Programming Languages and Software Engineering, University of Washington, Seattle, Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8816-6213","authenticated-orcid":false,"given":"Sergio","family":"Segura","sequence":"additional","affiliation":[{"name":"SCORE Lab, Universidad de Sevilla, Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9827-1834","authenticated-orcid":false,"given":"Antonio","family":"Ruiz-Cort\u00e9s","sequence":"additional","affiliation":[{"name":"SCORE Lab, Universidad de Sevilla, Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,11]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Beet. 2025. Beet Repository. Retrieved from https:\/\/github.com\/isa-group\/Beet"},{"key":"e_1_3_2_3_2","unstructured":"BDD. 2025. Chai Assertion Library. Retrieved from https:\/\/www.chaijs.com\/api\/bdd\/"},{"key":"e_1_3_2_4_2","unstructured":"Docker Hub. 2025. Daikon AGORA Docker Image. Retrieved from https:\/\/hub.docker.com\/r\/javalenzuela\/daikon_agora"},{"key":"e_1_3_2_5_2","unstructured":"GitHub. 2025. Daikon AGORA Repository. Retrieved from https:\/\/github.com\/JuanCarlosAlonsoValenzuela\/daikon_modified"},{"key":"e_1_3_2_6_2","unstructured":"Daikon. 2025. Daikon Developer Manual. New Formatting for Invariants. Retrieved from https:\/\/plse.cs.washington.edu\/daikon\/download\/doc\/developer.html#New-formatting-for-invariants"},{"key":"e_1_3_2_7_2","unstructured":"Daikon. 2025. Daikon Front Ends and Instrumentation. Retrieved from https:\/\/plse.cs.washington.edu\/daikon\/download\/doc\/daikon.html#Front-ends-and-instrumentation"},{"key":"e_1_3_2_8_2","unstructured":"GitHub. 2025. extJWNL (Extended Java WordNet Library) Repository. Retrieved from https:\/\/github.com\/extjwnl\/extjwnl"},{"key":"e_1_3_2_9_2","unstructured":"GitHub. 2025. GitHub API. Retrieved from https:\/\/developer.github.com\/v3\/"},{"key":"e_1_3_2_10_2","unstructured":"GitHub. 2025. GitLab Issue. Invalid URLs in the getApiV4ProjectsIdBadges Operation. Retrieved from https:\/\/gitlab.com\/gitlab-org\/gitlab\/-\/issues\/473603"},{"key":"e_1_3_2_11_2","unstructured":"GitHub. 2025. JSONMutator. Retrieved from https:\/\/github.com\/isa-group\/JSONmutator"},{"key":"e_1_3_2_12_2","unstructured":"OpenAPI. 2025. OpenAPI Specification. Retrieved from https:\/\/www.openapis.org"},{"key":"e_1_3_2_13_2","unstructured":"Postman. 2025. Postman API Platform. Retrieved from https:\/\/www.postman.com"},{"key":"e_1_3_2_14_2","unstructured":"GitHub. 2025. PostmanAssertify Repository. Retrieved from https:\/\/github.com\/JuanCarlosAlonsoValenzuela\/PostmanAssertify"},{"key":"e_1_3_2_15_2","unstructured":"RapidAPI. 2025. RapidAPI API Directory. Retrieved from https:\/\/rapidapi.com\/products\/api-hub\/"},{"key":"e_1_3_2_16_2","unstructured":"Zenodo. 2025. Replication Package. Retrieved from https:\/\/doi.org\/10.5281\/zenodo.12506791"},{"key":"e_1_3_2_17_2","unstructured":"API. 2025. Repository APIs.guru. Retrieved from https:\/\/apis.guru"},{"key":"e_1_3_2_18_2","unstructured":"GitHub. 2025. RESTCountries API Issue. Empty IDD Value. Retrieved from https:\/\/gitlab.com\/restcountries\/restcountries\/-\/issues\/220#note_1726639875"},{"key":"e_1_3_2_19_2","unstructured":"GitHub. 2025. RESTCountries Commit Fixing Country with Negative Area Bug. Retrieved from https:\/\/gitlab.com\/restcountries\/restcountries\/-\/commit\/ee498c74ad21c93b66a577d63d2c8eacefc58d42"},{"key":"e_1_3_2_20_2","unstructured":"GitHub. 2025. RESTCountries GitLab Issue. Country with Negative Area. Retrieved from https:\/\/gitlab.com\/restcountries\/restcountries\/-\/issues\/219"},{"key":"e_1_3_2_21_2","unstructured":"GitHub. 2025. RESTCountries Response Fields. Retrieved from https:\/\/gitlab.com\/restcountries\/restcountries\/-\/blob\/master\/FIELDS.md"},{"key":"e_1_3_2_22_2","unstructured":"Spotify. 2025. Spotify Web API. Retrieved from https:\/\/developer.spotify.com\/web-api\/"},{"key":"e_1_3_2_23_2","unstructured":"SRC. 2025. SRC Grand Finalists 2023. Retrieved from https:\/\/src.acm.org\/grand-finalists\/2023"},{"key":"e_1_3_2_24_2","unstructured":"Maven Repository. 2025. Swagger Request Validator. Retrieved from https:\/\/mvnrepository.com\/artifact\/com.atlassian.oai\/swagger-request-validat"},{"key":"e_1_3_2_25_2","unstructured":"Vimeo. 2025. Vimeo API. Working with Authentication. Retrieved from https:\/\/developer.vimeo.com\/api\/authentication"},{"key":"e_1_3_2_26_2","unstructured":"VISA. 2025. Visa Developer Center. Retrieved from https:\/\/developer.visa.com"},{"key":"e_1_3_2_27_2","unstructured":"Postman. 2025. Write API Test Scripts in Postman. Retrieved from https:\/\/learning.postman.com\/docs\/writing-scripts\/test-scripts\/"},{"key":"e_1_3_2_28_2","unstructured":"Google. 2025. YouTube API. List Videos Operation. Retrieved from https:\/\/developers.google.com\/youtube\/v3\/docs\/videos\/list"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3120680"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3559080"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3150618"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598114"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00083"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00046"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473920"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510047"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213872"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111041"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00036"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468626"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2002951.2002955"},{"key":"e_1_3_2_45_2","unstructured":"Davide Corradini Michele Pasqua and Mariano Ceccato. 2023. Automated black-box testing of mass assignment vulnerabilities in RESTful APIs. arXiv.2301.01261. Retrieved from https:\/\/arxiv.org\/abs\/2301.01261"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1093\/logcom\/2.4.511"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510141"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00101"},{"key":"e_1_3_2_49_2","first-page":"24","volume-title":"Proceedings of the Workshop on Dynamic Analysis (WODA \u201903)","author":"Ernst Michael D.","year":"2003","unstructured":"Michael D. Ernst. 2003. Static and dynamic analysis: Synergy and duality. In Proceedings of the Workshop on Dynamic Analysis (WODA \u201903), 24\u201327."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.908957"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2007.01.015"},{"key":"e_1_3_2_52_2","volume-title":"Architectural Styles and the Design of Network-Based Software Architectures","author":"Roy Thomas Fielding","year":"2000","unstructured":"Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-Based Software Architectures. Ph.\u2009D. Dissertation. University of California, Irvine."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642989"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387903.3389309"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2013.6575318"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409719"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397374"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931061"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3617175"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180199"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00012"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3528637"},{"key":"e_1_3_2_64_2","unstructured":"Soneya Binta Hossain and Matthew Dwyer. 2024. TOGLL: Correct and strong test oracle generation with LLMs. arXiv:2405.03786. Retrieved from https:\/\/arxiv.org\/abs\/2405.03786"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2942921"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549086"},{"key":"e_1_3_2_67_2","volume-title":"Software Abstractions: Logic, Language, and Analysis","author":"Jackson Daniel","year":"2012","unstructured":"Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis. The MIT Press."},{"key":"e_1_3_2_68_2","volume-title":"APIs: A Strategy Guide","author":"Jacobson Daniel","year":"2011","unstructured":"Daniel Jacobson, Greg Brail, and Dan Woods. 2011. APIs: A Strategy Guide. O\u2019Reilly Media, Inc."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534406"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00023"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598131"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00218"},{"key":"e_1_3_2_73_2","unstructured":"Myeongsoo Kim Saurabh Sinha and Alessandro Orso. 2025. LlamaRestTest: Effective REST API testing with small language models. arXiv:2501.08598. Retrieved from https:\/\/arxiv.org\/abs\/2501.08598"},{"key":"e_1_3_2_74_2","unstructured":"Myeongsoo Kim Tyler Stennett Dhruv Shah Saurabh Sinha and Alessandro Orso. 2024. Leveraging large language models to improve REST API testing. arXiv:2312.00894. Retrieved from https:\/\/arxiv.org\/abs\/2312.00894"},{"key":"e_1_3_2_75_2","unstructured":"Myeongsoo Kim Tyler Stennett Saurabh Sinha and Alessandro Orso. 2024. A multi-agent approach for REST API testing with semantic graphs and LLM-driven inputs. arXiv:2411.07098. Retrieved from https:\/\/arxiv.org\/abs\/2411.07098"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534401"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534381"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510133"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.44"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00018"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00034"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00034"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65310-1_33"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3469082"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549144"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00041"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510120"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00084"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00112"},{"key":"e_1_3_2_90_2","volume-title":"Proceedings of the 1st Workshop on Runtime Verification (RV \u201901)","author":"Nimmer Jeremy W.","year":"2001","unstructured":"Jeremy W. Nimmer and Michael D. Ernst. 2001. Static verification of dynamically detected program invariants: Integrating Daikon and ESC\/Java. In Proceedings of the 1st Workshop on Runtime Verification (RV \u201901)."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/566172.566213"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13911-6_45"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608133"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","unstructured":"Mike Papadakis Marinos Kintis Jie Zhang Yue Jia Yves Le Traon and Mark Harman. 2019. Chapter six\u2014Mutation testing advances: An analysis and survey. In Advances in Computers. Atif M. Memon (Ed.) Vol. 112. Elsevier 275\u2013378. DOI: 10.1016\/bs.adcom.2018.03.015","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.29"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2017.2679715"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.5555\/2566876"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524846.3527338"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2764464"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678586"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409758"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C51114.2020.00089"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00024"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3191656"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380429"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510151"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00167"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1002\/stv.430"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510149"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3217544"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409716"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464940"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00022"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534387"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3726524","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T09:26:56Z","timestamp":1765531616000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3726524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,11]]},"references-count":114,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3726524"],"URL":"https:\/\/doi.org\/10.1145\/3726524","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,11]]},"assertion":[{"value":"2024-07-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}