{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T08:37:07Z","timestamp":1773736627505,"version":"3.50.1"},"reference-count":38,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>The decision tree test method works as a flowchart structure for conversational flow. It has predetermined questions and answers that guide the user through specific tasks. Inspired by principles of the decision tree test method in software engineering, this paper discusses intelligent AI test modeling chat systems, including basic concepts, quality validation, test generation and augmentation, testing scopes, approaches, and needs. The paper\u2019s novelty lies in an intelligent AI test modeling chatbot system built and implemented based on an innovative 3-dimensional AI test model for AI-powered functions in intelligent mobile apps to support model-based AI function testing, test data generation, and adequate test coverage result analysis. As a result, a case study is provided using a mental health and emotional intelligence chatbot system, Wysa. It helps in tracking and analyzing mood and helps in sentiment analysis.<\/jats:p>","DOI":"10.3390\/software4020012","type":"journal-article","created":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T08:56:55Z","timestamp":1747299415000},"page":"12","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["AI Testing for Intelligent Chatbots\u2014A Case Study"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1051-5839","authenticated-orcid":false,"given":"Jerry","family":"Gao","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, College of Engineering, San Jose State University, San Jose, CA 95192, USA"},{"name":"ALPSTouchStone, Inc., San Jose, CA 95134, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4601-2659","authenticated-orcid":false,"given":"Radhika","family":"Agarwal","sequence":"additional","affiliation":[{"name":"ALPSTouchStone, Inc., San Jose, CA 95134, USA"}]},{"given":"Prerna","family":"Garsole","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, College of Engineering, San Jose State University, San Jose, CA 95192, USA"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"key":"ref_1","unstructured":"Businesswire (2024, December 15). Global Chatbot Market Value to Increase by $1.11 Billion during 2020\u20132024|Business Continuity Plan and Forecast for the New Normal|Technavio. Available online: https:\/\/www.businesswire.com\/news\/home\/20201207005691\/en\/Global-Chatbot-Market-Value-to-Increase-by-1.11-Billion-during-2020-2024-Business-Continuity-Plan-and-Forecast-for-the-New-Normal-Technavio."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Ni, J., Young, T., Pandelea, V., Xue, F., Adiga, V., and Cambria, E. (2021). Recent Advances in Deep Learning-based Dialogue Systems. arXiv.","DOI":"10.1007\/s10462-022-10248-8"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"120164","DOI":"10.1109\/ACCESS.2019.2937107","article-title":"Testing and Quality Validation for AI Software\u2014Perspectives, Issues, and Practices","volume":"7","author":"Tao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gao, J., Garsole, P., Agarwal, R., and Liu, S. (2024, January 15\u201318). AI Test Modeling and Analysis for Intelligent Chatbot Mobile App\u2014 A Case Study on Wysa. Proceedings of the 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), Shanghai, China.","DOI":"10.1109\/AITest62860.2024.00024"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Vasconcelos, M., Candello, H., Pinhanez, C., and Santos, T.D. (2017, January 23\u201327). Bottester: Testing Conversational Systems with Simulated Users. Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, Joinville, Brazil.","DOI":"10.1145\/3160504.3160584"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Xing, Y., and Fern\u00e1ndez, R. (2018, January 5\u20138). Automatic Evaluation of Neural Personality-based Chatbots. Proceedings of the 11th International Conference on Natural Language Generation, Tilburg, The Netherlands.","DOI":"10.18653\/v1\/W18-6524"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Gaston, C., Kosmatov, N., and Le Gall, P. (2019). Testing Chatbots Using Metamorphic Relations. Testing Software and Systems, Springer. ICTSS, 2019; Lecture Notes in Computer, Science.","DOI":"10.1007\/978-3-030-31280-0"},{"key":"ref_8","unstructured":"Bravo-Santos, S., Guerra, E., de Lara, J., Shepperd, T.C.W.C.I.M., Abreu, F.B.E., da Silva, A.R., and P\u00e9rez-Castillo, R. (2020). Quality of Information and Communications Technology, Springer. QUATIC 2020; Communications in Computer and Information Science."},{"key":"ref_9","first-page":"e2313925121","article-title":"A Turing test of whether AI chatbots are behaviorally similar to humans","volume":"121","author":"Mei","year":"2024","journal-title":"Econ. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bozic, J., Tazl, O.A., and Wotawa, F. (2019, January 4\u20139). Chatbot Testing Using AI Planning. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Testing (AITest), Newark, CA, USA.","DOI":"10.1109\/AITest.2019.00-10"},{"key":"ref_11","unstructured":"Ruane, E., Faure, T., Smith, R., Bean, D., Carson-Berndsen, J., and Ventresque, A. (2018, January 7\u201311). BoTest: A Framework to Test the Quality of Conversational Agents Using Divergent Input Examples. Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, Tokyo, Japan."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Guichard, J., Ruane, E., Smith, R., Bean, D., and Ventresque, A. (2019, January 4\u20139). Assessing the Robustness of Conversational Agents using Paraphrases. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Testing (AITest), Newark, CA, USA.","DOI":"10.1109\/AITest.2019.000-7"},{"key":"ref_13","unstructured":"Kaleem, M., Alobadi, O., O\u2019Shea, J., and Crockett, K. (2016, January 28). Framework for the formulation of metrics for conversational agent evaluation. Proceedings of the RE-WOCHAT: Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents-Development and Evaluation Workshop Programme, Portoro\u017e, Slovenia."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Nick, M., and Tautz, C. (1999, January 3\u20135). Practical evaluation of an organizational memory using the goal-question-metric technique. Proceedings of the Biannual German Conference on Knowledge-Based Systems, W\u00fcrzburg, Germany.","DOI":"10.1007\/10703016_9"},{"key":"ref_15","unstructured":"Das, K.N., Bansal, J.C., Deep, K., Nagar, A.K., Pathipooranam, P., and Naidu, R.C. (2020). Test Path Identification for Virtual Assistants Based on a Chatbot Flow Specifications. Soft Computing for Problem Solving. Advances in Intelligent Systems and Computing, Springer."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1543","DOI":"10.1007\/s10462-022-10205-5","article-title":"Survey on reinforcement learning for language processing","volume":"56","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2379","DOI":"10.1093\/jamia\/ocae215","article-title":"A review of reinforcement learning for natural language processing and applications in healthcare","volume":"31","author":"Liu","year":"2024","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"62","DOI":"10.47672\/ejt.1561","article-title":"The impact of artificial intelligence on chatbot technology: A study on the current advancements and leading innovations","volume":"7","author":"Aslam","year":"2023","journal-title":"Eur. J. Technol."},{"key":"ref_19","unstructured":"Ayanouz, S., Abdelhakim, B.A., and Benhmed, M. (April, January 31). A smart chatbot architecture based NLP and machine learning for health care assistance. Proceedings of the 3rd International Conference on Networking, Information Systems, & Security, Marrakech, Morocco."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bialkova, S. (2024). Chatbot Efficiency\u2014-Model Testing. The Rise of AI User Applications, Springer.","DOI":"10.1007\/978-3-031-56471-0"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"9601630","DOI":"10.1155\/2022\/9601630","article-title":"Emotionally intelligent chatbots: A systematic literature review","volume":"2022","author":"Bilquise","year":"2022","journal-title":"Hum. Behav. Emerg. Technol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Caldarini, G., Jaf, S., and McGarry, K. (2022). A Literature Survey of Recent Advances in Chatbots. Information, 13.","DOI":"10.3390\/info13010041"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gao, J., Tao, C., Jie, D., and Lu, S. (2019, January 4\u20139). Invited Paper: What is AI Software Testing? and Why. Proceedings of the IEEE International Conference on Service-Oriented System Engineering (SOSE), San Francisco, CA, USA.","DOI":"10.1109\/SOSE.2019.00015"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gao, J., Patil, P.H., Lu, S., Cao, D., and Tao, C. (2021, January 23\u201326). Model-Based Test Modeling and Automation Tool for Intelligent Mobile Apps. Proceedings of the IEEE International Conference on Service-Oriented System Engineering (SOSE), Oxford, UK.","DOI":"10.1109\/SOSE52839.2021.00028"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gao, J., Li, S., Tao, C., He, Y., Anumalasetty, A.P., Joseph, E.W., and Nayani, H. (2022, January 15\u201318). An approach to GUI test scenario generation using machine learning. Proceedings of the 2022 IEEE International Conference on Artificial Intelligence Testing (AITest), Newark, CA, USA.","DOI":"10.1109\/AITest55621.2022.00020"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2302980","DOI":"10.1080\/07853890.2024.2302980","article-title":"A systematic review of artificial intelligence-powered (AI-powered) chatbot intervention for managing chronic illness","volume":"56","author":"Kurniawan","year":"2024","journal-title":"Ann. Med."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, X., Tao, C., Gao, J., and Guo, H. (2022, January 5\u201318). A Review of Quality Assurance Research of Dialogue Systems. Proceedings of the 2022 IEEE International Conference on Artificial Intelligence Testing (AITest), Newark, CA, USA.","DOI":"10.1109\/AITest55621.2022.00021"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lin, C.-C., Huang, A.Y.Q., and Yang, S.J.H. (2023). A Review of AI-Driven Conversational Chatbots Implementation Methodologies and Challenges (1999\u20132022). Sustainability, 15.","DOI":"10.3390\/su15054012"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"101098","DOI":"10.1016\/j.elerap.2021.101098","article-title":"An intelligent knowledge-based chatbot for customer service","volume":"50","author":"Ngai","year":"2021","journal-title":"Electron. Commer. Res. Appl."},{"key":"ref_30","first-page":"17","article-title":"Application of AI based Chatbot Technology in the Industry","volume":"25","author":"Park","year":"2020","journal-title":"J. Korea Soc. Comput. Inf."},{"key":"ref_31","first-page":"26","article-title":"Systematic review on chatbot techniques and applications","volume":"18","author":"Park","year":"2022","journal-title":"J. Inf. Process. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"012080","DOI":"10.1088\/1742-6596\/1828\/1\/012080","article-title":"An intelligent mobile application testing experience report","volume":"1828","author":"Tran","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_33","first-page":"e27850","article-title":"Chatbot for health care and oncology applications using artificial intelligence and machine learning: Systematic review","volume":"7","author":"Xu","year":"2021","journal-title":"J. Med. Internet Res. (JMIR) Cancer"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1108\/IJPPM-10-2018-0366","article-title":"Understanding the components of profitability and productivity change at the micro level","volume":"69","author":"Wiech","year":"2020","journal-title":"Int. J. Product. Perform. Manag."},{"key":"ref_35","unstructured":"Chung, I.S., Malcolm, M., Lee, W.K., and Kwon, Y.R. (1996, January 19\u201323). Applying conventional testing techniques for class testing. Proceedings of the 20th International Computer Software and Applications Conference: COMPSAC\u201996, Seoul, Republic of Korea."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Pacheco, C., Lahiri, S.K., Ernst, M.D., and Ball, T. (2007, January 20\u201326). Feedback-Directed Random Test Generation. Proceedings of the 29th International Conference on Software Engineering (ICSE\u201907), Minneapolis, MN, USA.","DOI":"10.1109\/ICSE.2007.37"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/MS.2012.13","article-title":"Model-Based Testing","volume":"29","author":"Schieferdecker","year":"2012","journal-title":"IEEE Softw."},{"key":"ref_38","unstructured":"Freitas, T.C., Neto, A.C., Pereira, M.J.V., and Henriques, P.R. (2023, January 26\u201328). NLP\/AI Based Techniques for Programming Exercises Generation. Proceedings of the 4th International Computer Programming Education Conference (ICPEC 2023), Vila do Conde, Portugal."}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/4\/2\/12\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:33:17Z","timestamp":1760031197000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/4\/2\/12"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,15]]},"references-count":38,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["software4020012"],"URL":"https:\/\/doi.org\/10.3390\/software4020012","relation":{},"ISSN":["2674-113X"],"issn-type":[{"value":"2674-113X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,15]]}}}