{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T15:07:55Z","timestamp":1784300875769,"version":"3.55.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100031564","name":"Kristiania University College","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100031564","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Autom Softw Eng"],"published-print":{"date-parts":[[2025,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Testing is an important task within software development. To write test cases and integrate them into an automated test suite requires a significant amount of work. Given a set of requirements and specifications of a software, testing is needed to verify its correctness. When done manually, it is an expensive and error prone task. To facilitate such work, automated test-case generation via tools could be useful. Test-case generation can be facilitated by deterministic algorithm-driven approaches or non-deterministic approaches such as with AI (e.g., evolutionary and LLM). The different approaches come with their strengths and weaknesses, which must be considered when integrating these approaches into a product test procedure in industry. Several novel testing techniques and tools have been developed in academia and industry, but how effective they are and how to integrate them in real-world large industrial scenarios is still unclear. In this paper, a systematic approach is presented to evaluate test-case generation methodologies and integrate them into a scalable enterprise setup. The specific context is black-box testing of REST APIs, based on their OpenAPI schemas. The aim is to facilitate IT product development and service delivery. The proposed Technology Adoption Performance Evaluation (TAPE) approach is evaluated by a case study within the Group IT of Volkswagen AG. We evaluated existing tools such as OpenAPI Generator, EvoMaster and StarCoder which are built on different technologies. Our results show that these tools are of benefit for test engineers to facilitate test-case specification and design within the Group IT of Volkswagen AG.<\/jats:p>","DOI":"10.1007\/s10515-024-00477-2","type":"journal-article","created":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T05:49:46Z","timestamp":1733118586000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Technology adoption performance evaluation applied to testing industrial REST APIs"],"prefix":"10.1007","volume":"32","author":[{"given":"Alexander","family":"Poth","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olsi","family":"Rrjolli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Arcuri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,2]]},"reference":[{"key":"477_CR1","unstructured":"Apache JMeter: https:\/\/github.com\/apache\/jmeter (link validated 2024\/02\/05)"},{"issue":"3","key":"477_CR2","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MS.2020.3013820","volume":"38","author":"A Arcuri","year":"2020","unstructured":"Arcuri, A.: Automated black-and white-box testing of restful APIS with evomaster. IEEE Softw. 38(3), 72\u201378 (2020)","journal-title":"IEEE Softw."},{"issue":"8","key":"477_CR3","doi-asserted-by":"crossref","first-page":"1323","DOI":"10.1108\/00251740910984578","volume":"47","author":"A Baregheh","year":"2009","unstructured":"Baregheh, A., Rowley, J., Sambrook, S.: Towards a multidisciplinary definition of innovation. Manag. Decis. 47(8), 1323\u20131339 (2009)","journal-title":"Manag. Decis."},{"key":"477_CR4","doi-asserted-by":"crossref","unstructured":"Biagiola, M., Klikovits, S., Peltom\u00e4ki, J.. Riccio, V.: Sbft tool competition 2023-cyber-physical systems track. In 2023 IEEE\/ACM International Workshop on Search-Based and Fuzz Testing (SBFT), pp. 45\u201348. IEEE (2023).","DOI":"10.1109\/SBFT59156.2023.00010"},{"key":"477_CR5","unstructured":"ChatGPT data use: https:\/\/www.techradar.com\/news\/samsung-workers-leaked-company-secrets-by-using-chatgpt (link validated 2024\/02\/05)"},{"key":"477_CR6","unstructured":"ChatGPT issues: https:\/\/www.businessinsider.com\/chatgpt-companies-issued-bans-restrictions-openai-ai-amazon-apple-2023-7 (checked 2023\/10\/05)"},{"key":"477_CR7","unstructured":"Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G. et al.: Evaluating large language models trained on code. (2021). arXiv preprint arXiv:2107.03374 (2021)."},{"issue":"4","key":"477_CR8","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1023\/A:1011285919106","volume":"10","author":"BJ Copeland","year":"2000","unstructured":"Copeland, B.J.: The turing test. Mind. Mach. 10(4), 519\u2013539 (2000)","journal-title":"Mind. Mach."},{"key":"477_CR9","unstructured":"EU AI Act announcement (link validated 2024\/02\/05) https:\/\/www.europarl.europa.eu\/news\/en\/headlines\/society\/20230601STO93804\/eu-ai-act-first-regulation-on-artificial-intelligence"},{"key":"477_CR10","unstructured":"EvoMaster: https:\/\/github.com\/EMResearch\/EvoMaster (link validated 2024\/02\/05)"},{"key":"477_CR11","doi-asserted-by":"crossref","unstructured":"Gambi, A., Jahangirova, G., Riccio, V., Zampetti, F.: SBST tool competition 2022. In: Proceedings of the 15th Workshop on Search-Based Software Testing, pp. 25\u201332 (2022).","DOI":"10.1145\/3526072.3527538"},{"issue":"8","key":"477_CR12","first-page":"e2251","volume":"32","author":"V Garousi","year":"2020","unstructured":"Garousi, V., Felderer, M., Kuhrmann, M., Herkilo \u0306glu, M., Eldh, M.: Exploring the industry\u2019s challenges in software testing: an empirical study. J. Software: Evolut. Process 32(8), e2251 (2020)","journal-title":"J. Software: Evolut. Process"},{"key":"477_CR13","unstructured":"OpenAPI Generator: https:\/\/github.com\/OpenAPITools\/openapi-generator (checked 2023\/10\/5)"},{"key":"477_CR14","doi-asserted-by":"crossref","unstructured":"Golmohammadi, A., Zhang, M. and Arcuri, A., 2022. Testing RESTful APIs: A Survey. ACM Transactions on Software Engineering and Methodology.","DOI":"10.1145\/3617175"},{"key":"477_CR15","doi-asserted-by":"crossref","unstructured":"Hamid, O.H.: From model-centric to data-centric AI: A paradigm shift or rather a complementary approach?. In: 2022 8th International Conference on Information Technology Trends (ITT), pp. 196\u2013199. IEEE (2022).","DOI":"10.1109\/ITT56123.2022.9863935"},{"key":"477_CR16","doi-asserted-by":"crossref","unstructured":"Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1\u201312. IEEE (2015).","DOI":"10.1109\/ICST.2015.7102580"},{"key":"477_CR17","unstructured":"Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D. et al.: Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)"},{"key":"477_CR18","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1023\/A:1008367325700","volume":"9","author":"J Hernandez-Orallo","year":"2000","unstructured":"Hernandez-Orallo, J.: Beyond the Turing test. J. Logic Lang. Inform. 9, 447\u2013466 (2000)","journal-title":"J. Logic Lang. Inform."},{"issue":"3","key":"477_CR19","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1198\/106186005X59630","volume":"14","author":"T Hothorn","year":"2005","unstructured":"Hothorn, T., Leisch, F., Zeileis, A., Hornik, K.: The design and analysis of benchmark experiments. J. Comput. Graph. Stat. 14(3), 675\u2013699 (2005)","journal-title":"J. Comput. Graph. Stat."},{"key":"477_CR20","doi-asserted-by":"crossref","unstructured":"Huppler, K.: The art of building a good benchmark. In: Technology conference on performance evaluation and benchmarking, pp. 18\u201330. Berlin, Heidelberg: Springer Berlin Heidelberg (2009).","DOI":"10.1007\/978-3-642-10424-4_3"},{"key":"477_CR21","unstructured":"ISTQB glossary: https:\/\/istqb-glossary.page\/ (link validated 2024\/02\/05)"},{"key":"477_CR22","unstructured":"JaCoCo: https:\/\/github.com\/jacoco\/jacoco (link validated 2024\/02\/05)"},{"key":"477_CR23","doi-asserted-by":"crossref","unstructured":"Kistowski, J., Arnold, J.A., Huppler, K., Lange, K.D., Henning, J.L., Cao, P.: How to build a benchmark. In; Proceedings of the 6th ACM\/SPEC international conference on performance engineering, pp. 333\u2013336 (2015).","DOI":"10.1145\/2668930.2688819"},{"key":"477_CR24","unstructured":"LangChain: https:\/\/js.langchain.com\/docs\/get_started\/introduction (link validated 2024\/02\/05)"},{"key":"477_CR25","doi-asserted-by":"crossref","unstructured":"Lemieux, C., Inala, J.P., Lahiri, S.K., Sen, S. CODAMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In: International conference on software engineering (ICSE) (2023).","DOI":"10.1109\/ICSE48619.2023.00085"},{"issue":"6624","key":"477_CR26","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1126\/science.abq1158","volume":"378","author":"Y Li","year":"2022","unstructured":"Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al.: Competition-level code generation with alphacode. Science 378(6624), 1092\u20131097 (2022)","journal-title":"Science"},{"key":"477_CR27","unstructured":"Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210 (2023)."},{"key":"477_CR28","first-page":"36","volume":"13","author":"A Madaan","year":"2024","unstructured":"Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S.: Self-refine: iterative refinement with self-feedback. Adv. Neural. Inf. Process. Syst. 13, 36 (2024)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"issue":"13","key":"477_CR29","first-page":"9","volume":"174","author":"RK Mallidi","year":"2021","unstructured":"Mallidi, R.K., Sharma, M.: Study on agile story point estimation techniques and challenges. Int. J. Comput. Appl. 174(13), 9\u201314 (2021)","journal-title":"Int. J. Comput. Appl."},{"key":"477_CR30","unstructured":"Mazumder, M., Banbury, C., Yao, X., Karla\u0161, B., Rojas, W.G., Diamos, S., Diamos, G., He, L., Parrish, A., Kirk, H.R., Quaye, J.: Dataperf: Benchmarks for data-centric AI development. arXiv preprint arXiv:2207.10062 (2022)."},{"key":"477_CR31","doi-asserted-by":"crossref","unstructured":"McMinn, P.: Search-based software testing: Past, present and future. In: 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, pp. 153\u2013163. IEEE (2011).","DOI":"10.1109\/ICSTW.2011.100"},{"key":"477_CR32","unstructured":"OpenAPI: https:\/\/swagger.io\/specification\/ (link validated 2024\/02\/05)"},{"key":"477_CR33","unstructured":"Ouyang, S., Zhang, J.M., Harman, M. and Wang, M.: LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828 (2023)."},{"key":"477_CR34","doi-asserted-by":"crossref","unstructured":"Poth, A., Heimann, C.: How to innovate software quality assurance and testing in large enterprises?. In: Systems, Software and Services Process Improvement: 25th European Conference, EuroSPI 2018, Bilbao, Spain, September 5\u20137, 2018, Proceedings 25, pp. 437\u2013442. Springer International Publishing (2018).","DOI":"10.1007\/978-3-319-97925-0_37"},{"key":"477_CR35","doi-asserted-by":"crossref","unstructured":"Poth, A., Heimann, C.: Frugal innovation approaches combined with an agile organization to establish an innovation value stream. In: European Conference on Software Process Improvement, pp. 260\u2013274. Cham: Springer Nature Switzerland (2023).","DOI":"10.1007\/978-3-031-42310-9_19"},{"key":"477_CR36","doi-asserted-by":"crossref","unstructured":"Poth, A., Rrjolli, O., Riel, A.: Integration-and system-testing aligned with cloud-native approaches for DevOps. In: 2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), pp. 201\u2013208. IEEE (2022).","DOI":"10.1109\/QRS-C57518.2022.00038"},{"key":"477_CR37","doi-asserted-by":"crossref","unstructured":"Poth, A., Wildegger, A., Levien, D.A.: Considerations about integration of GenAI into products and services from an ethical and legal perspective. In: European Conference on Software Process Improvement, pp. 155\u2013171. Cham: Springer Nature Switzerland (2024).","DOI":"10.1007\/978-3-031-71142-8_12"},{"key":"477_CR38","doi-asserted-by":"crossref","unstructured":"Ren, X., Ye, X., Zhao, D., Xing, Z., Yang, X.: From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining. arXiv preprint arXiv:2309.15606 (2023).","DOI":"10.1109\/ASE56229.2023.00143"},{"key":"477_CR39","unstructured":"StarCoder: https:\/\/huggingface.co\/bigcode\/starcoder (link validated 2024\/02\/05)"},{"key":"477_CR40","doi-asserted-by":"crossref","unstructured":"Tang, Y., Liu, Z., Zhou, Z., Luo, X.: ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. arXiv preprint arXiv:2307.00588 (2023).","DOI":"10.1109\/TSE.2024.3382365"},{"key":"477_CR41","unstructured":"Willcocks, L., Hindle, J., Lacity, M.: Keys to RPA success. Executive Res. Rep., Knowl. Capital Partners, USA, Tech. Rep (2018)."},{"issue":"6","key":"477_CR42","first-page":"1","volume":"32","author":"M Zhang","year":"2023","unstructured":"Zhang, M., Arcuri, A.: Open problems in fuzzing restful APIS: a comparison of tools. ACM Trans. Softw. Eng. Methodol. 32(6), 1\u201345 (2023)","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"477_CR43","unstructured":"Zhang, M., Arcuri, A., Li, Y., Xue, K., Wang, Z., Huo, J., Huang, W.: Fuzzing microservices in industry: Experience of applying evomaster at meituan. arXiv preprint arXiv:2208.03988 (2022)."}],"container-title":["Automated Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-024-00477-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10515-024-00477-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-024-00477-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,6]],"date-time":"2025-04-06T01:39:19Z","timestamp":1743903559000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10515-024-00477-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,2]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,5]]}},"alternative-id":["477"],"URL":"https:\/\/doi.org\/10.1007\/s10515-024-00477-2","relation":{},"ISSN":["0928-8910","1573-7535"],"issn-type":[{"value":"0928-8910","type":"print"},{"value":"1573-7535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,2]]},"assertion":[{"value":"2 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"5"}}