{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T14:34:39Z","timestamp":1787236479702,"version":"build-2736575974"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T00:00:00Z","timestamp":1787184000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T00:00:00Z","timestamp":1787184000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004593","name":"Universidad Aut\u00f3noma de Madrid","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004593","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Conversational agents\u2014or chatbots\u2014are increasingly being used to access all sorts of services, like citizen services in city halls, customer support, or shopping. Moreover, recent advances in generative artificial intelligence are prompting the integration of conversational assistants into many applications, like programming IDEs, office automation software, or operating systems. Given the prominence of these agents, their correctness is a rising concern. However, automated and robust testing techniques for conversational systems are still needed. In this paper, we present a reverse engineering technique for extracting a behavioural model of a deployed chatbot (i.e., treated as a black-box) through the automated exploration of its functionality via Large Language Models. The extracted model is used for automated testing by generating testing conversation profiles, which a user simulator employs to conduct focused conversations with the chatbot-under-test. We describe our supporting tool\n                    <jats:sc>Tracer<\/jats:sc>\n                    , available as an integrated chatbot testing web platform at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"http:\/\/miso.ii.uam.es:8081\/\" ext-link-type=\"uri\">http:\/\/miso.ii.uam.es:8081\/<\/jats:ext-link>\n                    . We report on three evaluations that show that our exploration technique can yield an accurate model of the chatbot-under-test (via coverage analysis), the subsequent testing can discover existing errors in the chatbot (via mutation testing), and the exploration is precise in discovering functionality in real-world chatbots in production built with heterogeneous technologies.\n                  <\/jats:p>","DOI":"10.1007\/s42979-026-05237-5","type":"journal-article","created":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T13:33:12Z","timestamp":1787232792000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Reverse Engineering of Chatbot Behaviour for Automated Testing"],"prefix":"10.1007","volume":"7","author":[{"given":"Iv\u00e1n Sotillo","family":"del Horno","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandro","family":"del Pozzo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Esther","family":"Guerra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9425-6362","authenticated-orcid":false,"given":"Juan","family":"de Lara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,8,20]]},"reference":[{"key":"5237_CR1","unstructured":"OpenAI. https:\/\/openai.com\/. Accessed 2026."},{"key":"5237_CR2","unstructured":"Google: Gemini. https:\/\/gemini.google.com\/. Accessed 2026."},{"issue":"4","key":"5237_CR3","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1109\/MS.2020.3030198","volume":"38","author":"S P\u00e9rez-Soler","year":"2021","unstructured":"P\u00e9rez-Soler S, Ju\u00e1rez-Puerta S, Guerra E, Lara J. Choosing a chatbot development tool. IEEE Softw. 2021;38(4):94\u2013103.","journal-title":"IEEE Softw"},{"key":"5237_CR4","unstructured":"Google\u2019s Dialogflow. https:\/\/cloud.google.com\/dialogflow. Accessed 2026."},{"key":"5237_CR5","unstructured":"Rasa. https:\/\/rasa.com\/. Accessed 2026."},{"key":"5237_CR6","unstructured":"LangGraph. https:\/\/www.langchain.com\/langgraph. Accessed 2026."},{"key":"5237_CR7","unstructured":"Crew AI. https:\/\/www.crewai.com\/ Accessed (2026)."},{"key":"5237_CR8","unstructured":"Microsoft\u2019s AutoGen. https:\/\/microsoft.github.io\/autogen\/stable\/. Accessed 2026."},{"key":"5237_CR9","doi-asserted-by":"crossref","unstructured":"S\u00e1nchez Cuadrado J, P\u00e9rez-Soler S, Guerra E, de Lara J. Automating the development of task-oriented LLM-based chatbots. In: Proceedings of the 2024 ACM conference on conversational user interfaces, CUI. New York: ACM; 2024. p. 1\u201317.","DOI":"10.1145\/3640794.3665538"},{"key":"5237_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2026.101401","volume":"87","author":"\u00c1 Quiles","year":"2026","unstructured":"Quiles \u00c1, Guerra E, de Lara J. An intelligent low-code platform for building task-oriented LLM-based chatbots. J Comput Lang. 2026;87:101401.","journal-title":"J Comput Lang"},{"issue":"4","key":"5237_CR11","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1145\/3637228","volume":"33","author":"PC Ca\u00f1izares","year":"2024","unstructured":"Ca\u00f1izares PC, L\u00f3pez-Morales JM, P\u00e9rez-Soler S, Guerra E, de Lara J. Measuring and clustering heterogeneous chatbot designs. ACM Trans Softw Eng Methodol. 2024;33(4):90\u201319043.","journal-title":"ACM Trans Softw Eng Methodol"},{"issue":"5","key":"5237_CR12","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/MS.2024.3401551","volume":"41","author":"J S\u00e1nchez Cuadrado","year":"2024","unstructured":"S\u00e1nchez Cuadrado J, Avila Ortiz RD, P\u00e9rez-Soler S, Ca\u00f1izares PC, Guerra E, de Lara J. Integrating static quality assurance in CI chatbot development workflows. IEEE Softw. 2024;41(5):60\u20139.","journal-title":"IEEE Softw"},{"key":"5237_CR13","unstructured":"Rasa test. https:\/\/rasa.com\/docs\/pro\/testing\/evaluating-assistant\/. Accessed 2026."},{"key":"5237_CR14","unstructured":"Botium. https:\/\/cyara.com\/products\/botium\/. Accessed 2026."},{"issue":"11&12","key":"5237_CR15","doi-asserted-by":"publisher","first-page":"1673","DOI":"10.1142\/S0218194019400163","volume":"29","author":"R Ren","year":"2019","unstructured":"Ren R, Castro JW, Acu\u00f1a ST, de Lara J. Evaluation techniques for chatbot usability: a systematic mapping study. Int J Softw Eng Knowl Eng. 2019;29(11&12):1673\u2013702.","journal-title":"Int J Softw Eng Knowl Eng"},{"key":"5237_CR16","doi-asserted-by":"crossref","unstructured":"Vasconcelos M, Candello H, Pinhanez C, Santos T. Bottester: testing conversational systems with simulated users. In: Proceedings of the XVI Brazilian symposium on human factors in computing systems, IHC. New York: ACM; 2017. p. 1734\u201373.","DOI":"10.1145\/3160504.3160584"},{"key":"5237_CR17","doi-asserted-by":"crossref","unstructured":"Ca\u00f1izares PC, Avila Ortiz RD, P\u00e9rez-Soler S, Guerra E, de Lara J. Coverage-based strategies for the automated synthesis of test scenarios for conversational agents. In: Proceedings of the 5th ACM\/IEEE International Conference on Automation of Software Test, AST. New York: ACM; 2024. pp. 23\u201333.","DOI":"10.1145\/3644032.3644456"},{"key":"5237_CR18","doi-asserted-by":"crossref","unstructured":"G\u00f3mez-Abajo P, P\u00e9rez-Soler S, Ca\u00f1izares PC, Guerra E, de Lara J. Mutation testing for task-oriented chatbots. In: Proceedings of the 28th international conference on evaluation and assessment in software engineering, EASE. New York: ACM; 2024. pp. 232\u201341.","DOI":"10.1145\/3661167.3661220"},{"key":"5237_CR19","doi-asserted-by":"crossref","unstructured":"Urrico MF, Clerissi D, Mariani L. MutaBot: a mutation testing approach for chatbots. In: Proceedings of the 2024 IEEE\/ACM 46th international conference on software engineering: companion proceedings, ICSE companion. New York: ACM; 2024. p. 79\u201383.","DOI":"10.1145\/3639478.3640032"},{"key":"5237_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/B978-0-12-408089-8.00001-X","volume":"91","author":"N Walkinshaw","year":"2013","unstructured":"Walkinshaw N. Reverse-engineering software behavior. Adv Comput. 2013;91:1\u201358.","journal-title":"Adv Comput"},{"key":"5237_CR21","doi-asserted-by":"crossref","unstructured":"de Lara J, del Pozzo A, Guerra E, S\u00e1nchez Cuadrado J. Automated end-to-end testing for conversational agents. J Syst Softw. 2026;233:112685. See also https:\/\/github.com\/sensei-chat\/sensei","DOI":"10.1016\/j.jss.2025.112685"},{"issue":"4","key":"5237_CR22","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/C-M.1978.218136","volume":"11","author":"RA DeMillo","year":"1978","unstructured":"DeMillo RA, Lipton RJ, Sayward FG. Hints on test data selection: help for the practicing programmer. Computer. 1978;11(4):34\u201341.","journal-title":"Computer"},{"key":"5237_CR23","doi-asserted-by":"crossref","unstructured":"Sotillo del Horno I, Pozzo A, Guerra E, de Lara J. Automated exploration of conversational agents for the synthesis of testing profiles. Proceedings of the 37th IFIP WG 6.1 international conference on testing software and systems, ICTSS. LNCS, vol. 16107. Cham: Springer; 2025. pp. 213\u201330.","DOI":"10.1007\/978-3-032-05188-2_14"},{"key":"5237_CR24","volume-title":"About face: the essentials of interaction design","author":"A Cooper","year":"2014","unstructured":"Cooper A, Reimann R, Cronin D, Noessel C. About face: the essentials of interaction design. 4th ed. Indianapolis: Wiley; 2014.","edition":"4"},{"key":"5237_CR25","unstructured":"Moore RJ, Arar R. Conversational UX design: a practitioner\u2019s guide to the natural conversation framework. ACM Books, vol. 27. New York: ACM; 2019."},{"key":"5237_CR26","volume-title":"Design patterns: elements of reusable object-oriented software","author":"E Gamma","year":"1994","unstructured":"Gamma E, Helm R, Johnson R, Vlissides JM. Design patterns: elements of reusable object-oriented software. 1st ed. Boston: Addison-Wesley Professional; 1994.","edition":"1"},{"key":"5237_CR27","volume-title":"Introduction to software testing","author":"P Ammann","year":"2013","unstructured":"Ammann P, Offutt J. Introduction to software testing. Cambridge: Cambridge University Press; 2013."},{"issue":"3","key":"5237_CR28","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1002\/(SICI)1099-1689(199909)9:3<167::AID-STVR185>3.0.CO;2-E","volume":"9","author":"J Miller","year":"1999","unstructured":"Miller J. Estimating the number of remaining defects after inspection. Softw Test Verif Reliab. 1999;9(3):167\u201389.","journal-title":"Softw Test Verif Reliab"},{"issue":"6","key":"5237_CR29","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1109\/32.852741","volume":"26","author":"LC Briand","year":"2000","unstructured":"Briand LC, El Emam K, Freimut BG, Laitenberger O. A comprehensive evaluation of capture-recapture models for estimating software defect content. IEEE Trans Softw Eng. 2000;26(6):518\u201340.","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"5237_CR30","doi-asserted-by":"publisher","first-page":"1512","DOI":"10.2307\/1941127","volume":"72","author":"MW Palmer","year":"1991","unstructured":"Palmer MW. Estimating species richness: the second-order jackknife reconsidered. Ecology. 1991;72(4):1512\u20133.","journal-title":"Ecology"},{"key":"5237_CR31","doi-asserted-by":"crossref","unstructured":"Aichernig BK, Mostowski W, Mousavi MR, Tappler M, Taromirad M. Model learning and model-based testing. In: Machine learning for dynamic software analysis: potentials and limits-International Dagstuhl Seminar 16172, revised papers, LNCS, vol. 11026. Cham: Springer; 2018. p. 74\u2013100.","DOI":"10.1007\/978-3-319-96562-8_3"},{"key":"5237_CR32","doi-asserted-by":"crossref","unstructured":"Hajipour H, Malinowski M, Fritz M. IReEn: reverse-engineering of black-box functions via iterative neural program synthesis. In: Proceedings of the international workshop on machine learning and principles and practice of knowledge discovery in databases, ECML PKDD, Part II. Communications in computer and information science, vol. 1525. Cham: Springer; 2021. p. 143\u201357.","DOI":"10.1007\/978-3-030-93733-1_10"},{"key":"5237_CR33","unstructured":"Menguy G. Black-box code analysis for reverse engineering through constraint acquisition and program synthesis. France; 2023. (Ph.D. thesis)."},{"key":"5237_CR34","doi-asserted-by":"crossref","unstructured":"Ferreira R, Semedo D, Magalh\u00e3es J. Multi-trait user simulation with adaptive decoding for conversational task assistants. In: Findings of the association for computational linguistics, EMNLP. Miami: ACL; 2024. p. 16105\u201330.","DOI":"10.18653\/v1\/2024.findings-emnlp.945"},{"key":"5237_CR35","doi-asserted-by":"crossref","unstructured":"Sekulic I, Lu L, Bedi NS, Crestani F. Simulating conversational search users with parameterized behavior. In: Proceedings of the 2024 annual international ACM SIGIR conference on research and development in information retrieval in the Asia Pacific Region, SIGIR-AP. New York: ACM; 2024. p. 72\u201381.","DOI":"10.1145\/3673791.3698425"},{"issue":"2","key":"5237_CR36","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/s10791-022-09404-z","volume":"25","author":"A Salle","year":"2022","unstructured":"Salle A, Malmasi S, Rokhlenko O, Agichtein E. CoSearcher: studying the effectiveness of conversational search refinement and clarification through user simulation. Inf Retr J. 2022;25(2):209\u201338.","journal-title":"Inf Retr J"},{"key":"5237_CR37","doi-asserted-by":"crossref","unstructured":"Kiesel J, Gohsen M, Mirzakhmedova N, Hagen M, Stein B. Simulating follow-up questions in conversational search. In: Proceedings of the 46th European conference on information retrieval, ECIR. LNCS, vol. 14609. Cham: Springer; 2024. p. 382\u201398.","DOI":"10.1007\/978-3-031-56060-6_25"},{"issue":"9","key":"5237_CR38","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1080\/08839514.2013.835230","volume":"27","author":"D Griol","year":"2013","unstructured":"Griol D, Carb\u00f3 J, Molina JM. An automatic dialog simulation technique to develop and evaluate interactive conversational agents. Appl Artif Intell. 2013;27(9):759\u201380.","journal-title":"Appl Artif Intell"},{"key":"5237_CR39","doi-asserted-by":"crossref","unstructured":"Dhole KD. KAUCUS: knowledge augmented user simulators for training language model assistants. In: Proceedings of the 1st workshop on simulating conversational intelligence in chat, SCI-CHAT. Pennsylvania: ACL; 2024. p. 53\u201365.","DOI":"10.18653\/v1\/2024.scichat-1.5"},{"key":"5237_CR40","unstructured":"Terragni S, Filipavicius M, Khau N, Guedes B, Manso A, Mathis R. In-context learning user simulators for task-oriented dialog systems. 2023. arXiv:2306.00774."},{"issue":"4","key":"5237_CR41","first-page":"443","volume":"45","author":"J Bandlamudi","year":"2024","unstructured":"Bandlamudi J, Mukherjee K, Agarwal P, Chaudhuri R, Pimplikar R, Dechu S, et al. Framework to enable and test conversational assistant for APIs and RPAs. AI Mag. 2024;45(4):443\u201356.","journal-title":"AI Mag"},{"key":"5237_CR42","doi-asserted-by":"crossref","unstructured":"Wit J. Leveraging large language models as simulated users for initial, low-cost evaluations of designed conversations. In: Proceedings of 7th international workshop on chatbot research and design, conversations. LNCS, vol. 14524. Cham: Springer; 2023. p. 77\u201393.","DOI":"10.1007\/978-3-031-54975-5_5"},{"issue":"5","key":"5237_CR43","first-page":"79","volume":"6","author":"P Rao","year":"2021","unstructured":"Rao P, Agnihotri M, Jayagopi DB. Improving asynchronous interview interaction with follow-up question generation. Int J Interact Multimed Artif Intell. 2021;6(5):79\u201389.","journal-title":"Int J Interact Multimed Artif Intell"},{"key":"5237_CR44","doi-asserted-by":"crossref","unstructured":"Yalla M, Sunil A. AI-driven conversational bot test automation using industry specific data cartridges. In: Proceedings of the IEEE\/ACM 1st international conference on automation of software test, AST@ICSE. 2020;105\u20137.","DOI":"10.1145\/3387903.3389306"},{"key":"5237_CR45","doi-asserted-by":"crossref","unstructured":"Rapisarda RG, Ginelli D, Clerissi D, Mariani L. Test case generation for dialogflow task-based chatbots. In: Proceedings of the 2025 IEEE international conference on software testing, verification and validation workshops. ICSTW; 2025. p. 93\u2013102.","DOI":"10.1109\/ICSTW64639.2025.10962471"},{"key":"5237_CR46","doi-asserted-by":"crossref","unstructured":"Bravo-Santos S, Guerra E, de Lara J. Testing chatbots with Charm. In: Proceedings of the 13th international conference on quality of information and communications technology, QUATIC. CCIS, vol. 1266:. Cham: Springer; 2020. p. 426\u201338.","DOI":"10.1007\/978-3-030-58793-2_34"},{"issue":"6","key":"5237_CR47","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3654438","volume":"33","author":"E Guglielmi","year":"2024","unstructured":"Guglielmi E, Rosa G, Scalabrino S, Bavota G, Oliveto R. Help them understand: testing and improving voice user interfaces. ACM Trans Softw Eng Methodol. 2024;33(6):1\u201333.","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"5237_CR48","doi-asserted-by":"crossref","unstructured":"Elvira T, Procko TT, Vonderhaar L, Ochoa O. Exploring testing methods for large language models. In: Proceedings of the 2024 international conference on machine learning and applications, ICMLA. 2024. p. 1152\u20137.","DOI":"10.1109\/ICMLA61862.2024.00177"},{"key":"5237_CR49","doi-asserted-by":"crossref","unstructured":"Raina V, Liusie A, Gales MJF. Is LLM-as-a-judge robust? Investigating universal adversarial attacks on zero-shot LLM assessment. In: Proceedings of the 2024 conference on empirical methods in natural language processing, EMNLP. Pennsylvania: Association for Computational Linguistics; 2024. p. 7499\u201317.","DOI":"10.18653\/v1\/2024.emnlp-main.427"}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-026-05237-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-026-05237-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-026-05237-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T13:33:50Z","timestamp":1787232830000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-026-05237-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,20]]},"references-count":49,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["5237"],"URL":"https:\/\/doi.org\/10.1007\/s42979-026-05237-5","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,20]]},"assertion":[{"value":"24 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Research involving human and\/or animals","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":3,"name":"Ethics","label":"Informed consent","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"709"}}