{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T06:59:09Z","timestamp":1794034749216,"version":"4.8.0"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"DOI":"10.13039\/501100005416","name":"Research Council of Norway","doi-asserted-by":"crossref","award":["No. 309175"],"award-info":[{"award-number":["No. 309175"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Healthcare Internet of Things (IoT) applications often integrate various third-party healthcare applications and medical devices through REST APIs, resulting in complex and interdependent networks of REST APIs. Oslo City\u2019s healthcare department collaborates with various industry partners to develop these applications, enriched with diverse REST APIs that evolve during the DevOps process to accommodate evolving needs such as new features, services, and devices. Oslo City\u2019s primary goal is to utilize automated solutions for continuous testing of REST APIs at each evolution stage to ensure dependability. Although the literature offers various automated REST API testing tools, their effectiveness in regression testing of the evolving REST APIs of healthcare IoT applications within a DevOps context remains undetermined. This article evaluates state-of-the-art and well-established REST API testing tools\u2014specifically, RESTest, EvoMaster, Schemathesis, RESTler, and RestTestGen\u2014for the regression testing of a real-world healthcare IoT application, considering failures, faults, coverage, regressions, and cost. We conducted experiments using all accessible REST APIs (17 APIs with 120 endpoints), and 14 releases evolved during DevOps. Overall, all tools generated tests leading to several failures, 18 potential faults, up to 84% coverage, and 23 regressions. Over 70% of tests generated by all tools fail to detect failures, resulting in significant overhead.<\/jats:p>","DOI":"10.1145\/3765744","type":"journal-article","created":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T15:36:30Z","timestamp":1757000190000},"page":"1-46","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["REST API Testing in DevOps: A Study on an Evolving Healthcare IoT Application"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5212-9787","authenticated-orcid":false,"given":"Hassan","family":"Sartaj","sequence":"first","affiliation":[{"name":"Simula Research Laboratory, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9979-3519","authenticated-orcid":false,"given":"Shaukat","family":"Ali","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8657-5691","authenticated-orcid":false,"given":"Julie","family":"Marie Gj\u00f8by","sequence":"additional","affiliation":[{"name":"Welfare Technologies Section, Oslo Kommune Helseetaten, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,5,13]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICAMIMIA60881.2023.10427702"},{"key":"e_1_3_3_3_2","unstructured":"AceAge. 2025. Karie Medicine Dispenser. Retrieved April 18 2025 from https:\/\/kariehealth.com\/"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47169-3_55"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3150618"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3137003.3137007"},{"key":"e_1_3_3_7_2","unstructured":"Apiary. 2024. Dredd \u2014 HTTP API Testing Framework. Retrieved April 20 2024 from https:\/\/github.com\/apiaryio\/dredd"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00046"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-023-09620-w"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00083"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00046"},{"key":"e_1_3_3_14_2","volume-title":"Software Testing Techniques","author":"Beizer Boris","year":"2003","unstructured":"Boris Beizer. 2003. Software Testing Techniques (2nd. ed.). Dreamtech Press.","edition":"2"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2298"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3381386"},{"key":"e_1_3_3_17_2","unstructured":"Cornutum. 2024. Tcases: A Model-Based Test Case Generator. Retrieved April 20 2024 from https:\/\/github.com\/Cornutum\/tcases"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00213"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM52516.2021.00035"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00063"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME55016.2022.00068"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1808"},{"key":"e_1_3_3_23_2","unstructured":"Alix Decrop Gilles Perrouin Mike Papadakis Xavier Devroey and Pierre-Yves Schobbens. 2024. You Can REST Now: Automated Specification Inference and Black-Box Testing of RESTful APIs with Large Language Models. arXiv:2402.05102. Retrieved from https:\/\/arxiv.org\/abs\/2402.05102"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00035"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDOC.2018.00031"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.988497"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2023.01.322"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i1.31995"},{"key":"e_1_3_3_29_2","unstructured":"Python Software Foundation. 2024. difflib\u2014Helpers for Computing Deltas. Retrieved July 10 2024 from https:\/\/docs.python.org\/3\/library\/difflib.html"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1869459.1869475"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1805"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409719"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397374"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","unstructured":"Amid Golmohammadi Man Zhang and Andrea Arcuri. 2023. Testing RESTful APIs: A survey. 33 1 Article 27 (Nov. 2023). 41 pages. \\(\\mathtt{DOI:}\\) 10.1145\/3617175","DOI":"10.1145\/3617175"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3579851"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3528637"},{"key":"e_1_3_3_37_2","unstructured":"Helsedirektoratet. 2024. National Welfare Technology Program. Retrieved May 05 2024 from https:\/\/www.helsedirektoratet.no\/tema\/velferdsteknologi"},{"issue":"2","key":"e_1_3_3_38_2","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm Sture","year":"1979","unstructured":"Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65\u201370.","journal-title":"Scandinavian Journal of Statistics"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10017-1"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58846.2023.00065"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00023"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09717-6"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598131"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00218"},{"key":"e_1_3_3_45_2","unstructured":"Myeongsoo Kim Saurabh Sinha and Alessandro Orso. 2025. LlamaRestTest: Effective REST API testing with small language models. arXiv:2501.08598. Retrieved from https:\/\/arxiv.org\/abs\/2501.08598"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534401"},{"key":"e_1_3_3_47_2","unstructured":"Kiss Peter. 2024. APIFuzzer\u2014HTTP API Testing Framework. Retrieved April 20 2024 from https:\/\/github.com\/KissPeter\/APIFuzzer"},{"key":"e_1_3_3_48_2","unstructured":"Oslo Kommune. 2024. Norwegian Health Authority. Retrieved May 05 2024 from https:\/\/www.oslo.kommune.no\/etater-foretak-og-ombud\/helseetaten\/"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3056505"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1049\/iet-sen.2017.0344"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00023"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510133"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491038"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00034"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2021.3050610"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340433.3342822"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65310-1_33"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3469082"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549144"},{"key":"e_1_3_3_60_2","unstructured":"Medido. 2025. Automatic Medicine Dispenser Medido. Retrieved April 18 2025 from https:\/\/medido.com\/en\/"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/DeepTest52559.2021.00008"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1134\/S0361768822080175"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3549206.3549321"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C57518.2022.00038"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/edoc49727.2020.00026"},{"issue":"7","key":"e_1_3_3_66_2","first-page":"46","article-title":"Pattern matching: The Gestalt approach","volume":"13","author":"Ratcliff John W.","year":"1988","unstructured":"John W. Ratcliff and David E. Metzener. 1988. Pattern matching: The Gestalt approach. Dr. Dobb\u2019s Journal 13, 7 (1988), 46.","journal-title":"Dr. Dobb\u2019s Journal"},{"key":"e_1_3_3_67_2","unstructured":"Responssenteret. 2025. Pilly SMS Medicine Dispenser. Retrieved April 18 2025 from https:\/\/responssenteret.no\/responsskolen\/brukere\/manualer-videoer\/Pilly.php"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708534"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-024-01223-8"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-66326-0_28"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613888"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.3311"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-020-00844-z"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3182528"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/52.3-4.591"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2014.74"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678586"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9575-4"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3127131"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534365"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510151"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.430"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597205"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3765744","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T17:26:34Z","timestamp":1778693194000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3765744"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,13]]},"references-count":82,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3765744"],"URL":"https:\/\/doi.org\/10.1145\/3765744","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,13]]},"assertion":[{"value":"2024-10-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}