{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T15:18:18Z","timestamp":1773415098928,"version":"3.50.1"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Spanish Ministry of Science","award":["TED2021-129381B-C21, PID2021-122270OB-I00, and RED2022-134647-T"],"award-info":[{"award-number":["TED2021-129381B-C21, PID2021-122270OB-I00, and RED2022-134647-T"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Conversational agents, or chatbots, have become popular to access all kind of software services. They provide an intuitive natural language interface for interaction, available from a wide range of channels including social networks, web pages, intelligent speakers or cars. In response to this demand, many chatbot development platforms and tools have emerged. However, they typically lack support to statically measure properties of the chatbots being built, as indicators of their size, complexity, quality or usability. Similarly, there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies.<\/jats:p>\n          <jats:p\/>\n          <jats:p>\n            To overcome this limitation, we propose a suite of 21 metrics for chatbot designs, as well as two clustering methods that help in grouping chatbots along their conversation topics and design features. Both the metrics and the clustering methods are defined on a neutral chatbot design language, becoming independent of the implementation platform. We provide automatic translations of chatbots defined on some major platforms into this neutral notation to perform the measurement and clustering. The approach is supported by our tool\n            <jats:sc>Asymob<\/jats:sc>\n            , which we have used to evaluate the metrics and the clustering methods over a set of 259 Dialogflow and Rasa chatbots from open-source repositories. The results open the door to incorporating the metrics within chatbot development processes for the early detection of quality issues, and to exploit clustering to organise large collections of chatbots into significant groups to ease chatbot comprehension, search and comparison.\n          <\/jats:p>","DOI":"10.1145\/3637228","type":"journal-article","created":{"date-parts":[[2023,12,13]],"date-time":"2023-12-13T11:40:03Z","timestamp":1702467603000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Measuring and Clustering Heterogeneous Chatbot Designs"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2084-1558","authenticated-orcid":false,"given":"Pablo C.","family":"Ca\u00f1izares","sequence":"first","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5848-6224","authenticated-orcid":false,"given":"Jose Mar\u00eda","family":"L\u00f3pez-Morales","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4558-7111","authenticated-orcid":false,"given":"Sara","family":"P\u00e9rez-Soler","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2818-2278","authenticated-orcid":false,"given":"Esther","family":"Guerra","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9425-6362","authenticated-orcid":false,"given":"Juan","family":"de Lara","sequence":"additional","affiliation":[{"name":"Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}]}],"member":"320","published-online":{"date-parts":[[2024,4,17]]},"reference":[{"issue":"8","key":"e_1_3_3_2_2","doi-asserted-by":"crossref","first-page":"3087","DOI":"10.1109\/TSE.2021.3078384","article-title":"A comparison of natural language understanding platforms for chatbots in software engineering","volume":"48","author":"Abdellatif Ahmad","year":"2022","unstructured":"Ahmad Abdellatif, Khaled Badran, Diego Costa, and Emad Shihab. 2022. A comparison of natural language understanding platforms for chatbots in software engineering. IEEE Trans. Software Eng. 48, 8 (2022), 3087\u20133102.","journal-title":"IEEE Trans. Software Eng."},{"key":"e_1_3_3_3_2","first-page":"24:1\u201324:4","volume-title":"Proc. 9th Hellenic Conf. on Artificial Intelligence","author":"Arshad Shumail","year":"2016","unstructured":"Shumail Arshad and Christos Tjortjis. 2016. Clustering software metric values extracted from C# code for maintainability assessment. In Proc. 9th Hellenic Conf. on Artificial Intelligence. ACM, 24:1\u201324:4."},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-319-42061-5_1","volume-title":"Proc. 12th Eur. Conf. on Modelling Foundations and Applications (LNCS)","author":"Babur \u00d6nder","year":"2016","unstructured":"\u00d6nder Babur, Loek Cleophas, and Mark van den Brand. 2016. Hierarchical clustering of metamodels for comparative analysis and visualization. In Proc. 12th Eur. Conf. on Modelling Foundations and Applications (LNCS), Vol. 9764. Springer, 3\u201318."},{"key":"e_1_3_3_5_2","first-page":"342","volume-title":"Proc. 28th Int. Conf. on Advanced Information Syst. Eng. (LNCS)","author":"Basciani Francesco","year":"2016","unstructured":"Francesco Basciani, Juri Di Rocco, Davide Di Ruscio, Ludovico Iovino, and Alfonso Pierantonio. 2016. Automated clustering of metamodel repositories. In Proc. 28th Int. Conf. on Advanced Information Syst. Eng. (LNCS), Vol. 9694. Springer, 342\u2013358."},{"key":"e_1_3_3_6_2","unstructured":"Botium. 2023. https:\/\/www.botium.ai\/ (2023)."},{"key":"e_1_3_3_7_2","first-page":"41","volume-title":"Proc. 31st IFIP WG 6.1 Int. Conf. on Testing Softw. and Syst. (LNCS)","author":"Bozic Josip","year":"2019","unstructured":"Josip Bozic and Franz Wotawa. 2019. Testing chatbots using metamorphic relations. In Proc. 31st IFIP WG 6.1 Int. Conf. on Testing Softw. and Syst. (LNCS), Vol. 11812. Springer, 41\u201355."},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1007\/978-3-030-58793-2_34","volume-title":"Proc. 13th Int. Conf. on Quality of Information and Communications Technology (CCIS)","author":"Bravo-Santos Sergio","year":"2020","unstructured":"Sergio Bravo-Santos, Esther Guerra, and Juan de Lara. 2020. Testing chatbots with charm. In Proc. 13th Int. Conf. on Quality of Information and Communications Technology (CCIS), Vol. 1266. Springer, 426\u2013438."},{"key":"e_1_3_3_9_2","doi-asserted-by":"crossref","first-page":"104047","DOI":"10.1016\/j.jml.2019.104047","article-title":"How many words do we read per minute? A review and meta-analysis of reading rate","volume":"109","author":"Brysbaert Marc","year":"2019","unstructured":"Marc Brysbaert. 2019. How many words do we read per minute? A review and meta-analysis of reading rate. Journal of Memory and Language 109 (2019), 104047.","journal-title":"Journal of Memory and Language"},{"key":"e_1_3_3_10_2","first-page":"1","volume-title":"Proc. 37th ACM\/SIGAPP Symposium on Applied Computing","author":"Ca\u00f1izares Pablo C.","year":"2022","unstructured":"Pablo C. Ca\u00f1izares, Sara P\u00e9rez-Soler, Esther Guerra, and Juan de Lara. 2022. Automating the measurement of heterogeneous chatbot designs. In Proc. 37th ACM\/SIGAPP Symposium on Applied Computing. ACM, 1\u20138."},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","unstructured":"D. Cer Y. Yang S.-yi Kong N. Hua N. Limtiaco R. S. John N. Constant M. Guajardo-C\u00e9spedes S. Yuan C. Tar B. Strope and R. Kurzweil. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).","DOI":"10.18653\/v1\/D18-2029"},{"key":"e_1_3_3_12_2","unstructured":"Chatbottest. 2023. https:\/\/chatbottest.com\/ (2023)."},{"issue":"5","key":"e_1_3_3_13_2","first-page":"545","article-title":"The linguistic accuracy of chatbots: Usability from an ESL perspective","volume":"34","author":"Coniam David","year":"2014","unstructured":"David Coniam. 2014. The linguistic accuracy of chatbots: Usability from an ESL perspective. Text & Talk 34, 5 (2014), 545\u2013567.","journal-title":"Text & Talk"},{"key":"e_1_3_3_14_2","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume":"1810","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs\/1810.04805 (2018), 16.","journal-title":"CoRR"},{"key":"e_1_3_3_15_2","unstructured":"Dialogflow. 2023. https:\/\/dialogflow.com\/ (2023)."},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1145\/3425174.3425230","volume-title":"Proc. 5th Brazilian Symposium on Systematic and Automated Softw. Testing","author":"Santos M\u00e1rcio Braga dos","year":"2020","unstructured":"M\u00e1rcio Braga dos Santos, Ana Paula Carvalho Cavalcanti Furtado, Sidney C. Nogueira, and Diogo Dantas Moreira. 2020. OggyBug: A test automation tool in chatbots. In Proc. 5th Brazilian Symposium on Systematic and Automated Softw. Testing. ACM, 79\u201387."},{"key":"e_1_3_3_17_2","first-page":"226","volume-title":"Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining","author":"Ester Martin","year":"1996","unstructured":"Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining. AAAI Press, 226\u2013231."},{"key":"e_1_3_3_18_2","volume-title":"Software Metrics - A Practical and Rigorous Approach (2nd ed.)","author":"Fenton Norman E.","year":"1996","unstructured":"Norman E. Fenton and Shari Lawrence Pfleeger. 1996. Software Metrics - A Practical and Rigorous Approach (2nd ed.). International Thomson."},{"key":"e_1_3_3_19_2","first-page":"15044","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023","author":"Finch Sarah E.","year":"2023","unstructured":"Sarah E. Finch, James D. Finch, and Jinho D. Choi. 2023. Don\u2019t forget your abc\u2019s: Evaluating the state-of-the-art in chat-oriented dialogue systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 15044\u201315071."},{"issue":"3","key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick.","volume":"32","author":"Flesch R.","year":"1948","unstructured":"R. Flesch. 1948. A new readability yardstick. J. of Applied Psychology 32, 3 (1948), 221.","journal-title":"J. of Applied Psychology"},{"key":"e_1_3_3_21_2","unstructured":"Gartner. 2022. Competitive Landscape: Conversational AI Platform Providers. https:\/\/info.kore.ai\/competitive-landscape-conversational-ai-platform-providers (2022)."},{"key":"e_1_3_3_22_2","unstructured":"Google. 2023. Bard. https:\/\/bard.google.com\/ (2023)."},{"key":"e_1_3_3_23_2","unstructured":"ISO 9241-11. 1998. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs). Part II Guidance on Usability. (1998)."},{"key":"e_1_3_3_24_2","unstructured":"ISO\/IEC 25010. 2011. ISO\/IEC 25010:2011 Systems and Software Engineering \u2014 Systems and Software Quality Requirements and Evaluation (SQuaRE) \u2014 System and Software Quality Models. (2011)."},{"key":"e_1_3_3_25_2","volume-title":"Algorithms for Clustering Data","author":"Jain Anil K.","year":"1988","unstructured":"Anil K. Jain and Richard C. Dubes. 1988. Algorithms for Clustering Data. Prentice-Hall, Englewood Cliffs, NJ, USA."},{"key":"e_1_3_3_26_2","first-page":"1545","volume-title":"Proc. 43rd Int. ACM SIGIR Conf. on Research and Development in Information Retrieval","author":"Jiang Jiepu","year":"2020","unstructured":"Jiepu Jiang and Naman Ahuja. 2020. Response quality in human-chatbot collaborative systems. In Proc. 43rd Int. ACM SIGIR Conf. on Research and Development in Information Retrieval. ACM, 1545\u20131548."},{"issue":"7","key":"e_1_3_3_27_2","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1016\/j.jss.2005.06.044","article-title":"MUDABlue: An automatic categorization system for open source repositories","volume":"79","author":"Kawaguchi Shinji","year":"2006","unstructured":"Shinji Kawaguchi, Pankaj K. Garg, Makoto Matsushita, and Katsuro Inoue. 2006. MUDABlue: An automatic categorization system for open source repositories. J. Syst. Softw. 79, 7 (2006), 939\u2013953.","journal-title":"J. Syst. Softw."},{"issue":"3","key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1016\/j.infsof.2006.10.017","article-title":"Semantic clustering: Identifying topics in source code","volume":"49","author":"Kuhn Adrian","year":"2007","unstructured":"Adrian Kuhn, St\u00e9phane Ducasse, and Tudor G\u00eerba. 2007. Semantic clustering: Identifying topics in source code. Inf. Softw. Technol. 49, 3 (2007), 230\u2013243.","journal-title":"Inf. Softw. Technol."},{"issue":"2","key":"e_1_3_3_29_2","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1080\/01638539809545028","article-title":"An introduction to latent semantic analysis","volume":"25","author":"Landauer Thomas K.","year":"1998","unstructured":"Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse Processes 25, 2\u20133 (1998), 259\u2013284.","journal-title":"Discourse Processes"},{"issue":"1","key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MS.2017.4541027","article-title":"Software bots","volume":"35","author":"Lebeuf Carlene","year":"2018","unstructured":"Carlene Lebeuf, Margaret-Anne D. Storey, and Alexey Zagalsky. 2018. Software bots. IEEE Softw. 35, 1 (2018), 18\u201323.","journal-title":"IEEE Softw."},{"key":"e_1_3_3_31_2","unstructured":"Lex. 2023. https:\/\/aws.amazon.com\/en\/lex\/ (2023)."},{"key":"e_1_3_3_32_2","first-page":"2122","volume-title":"Proc. 2016 Conf. on Empirical Methods in Natural Language Processing","author":"Liu Chia-Wei","year":"2016","unstructured":"Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proc. 2016 Conf. on Empirical Methods in Natural Language Processing. ACL, 2122\u20132132."},{"issue":"3","key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"967","DOI":"10.1007\/s10270-021-00929-3","article-title":"ModelSet: A dataset for machine learning in model-driven engineering","volume":"21","author":"L\u00f3pez Jos\u00e9 Antonio Hern\u00e1ndez","year":"2022","unstructured":"Jos\u00e9 Antonio Hern\u00e1ndez L\u00f3pez, Javier Luis C\u00e1novas Izquierdo, and Jes\u00fas S\u00e1nchez Cuadrado. 2022. ModelSet: A dataset for machine learning in model-driven engineering. Softw. Syst. Model. 21, 3 (2022), 967\u2013986.","journal-title":"Softw. Syst. Model."},{"key":"e_1_3_3_34_2","first-page":"1","volume-title":"Proc. 44th Int. Conf. on Soft. Eng.","author":"L\u00f3pez-Morales Jos\u00e9-Mar\u00eda","year":"2022","unstructured":"Jos\u00e9-Mar\u00eda L\u00f3pez-Morales, Pablo C. Ca\u00f1izares, Sara P\u00e9rez-Soler, Esther Guerra, and Juan de Lara. 2022. Asymob: A platform for measuring and clustering chatbots. In Proc. 44th Int. Conf. on Soft. Eng.ACM, 1\u20135."},{"key":"e_1_3_3_35_2","first-page":"46","volume-title":"Proc. 12th IEEE Int. Conf. on Tools with Artificial Intelligence","author":"Maletic Jonathan I.","year":"2000","unstructured":"Jonathan I. Maletic and Andrian Marcus. 2000. Using latent semantic analysis to identify similarities in source code to support program understanding. In Proc. 12th IEEE Int. Conf. on Tools with Artificial Intelligence. IEEE CS, 46\u201353."},{"key":"e_1_3_3_36_2","volume-title":"Advances in Neural Information Processing Systems","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, Vol. 26. Curran Associates, Inc."},{"key":"e_1_3_3_37_2","first-page":"1786","volume-title":"Proc. 9th Int. Conf. on Spoken Language Processing","author":"M\u00f6ller Sebastian","year":"2006","unstructured":"Sebastian M\u00f6ller, Roman Englert, Klaus-Peter Engelbrecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Reithinger. 2006. Memo: Towards automatic usability evaluation of spoken dialogue services by user error simulations. In Proc. 9th Int. Conf. on Spoken Language Processing. ISCA, 1786\u20131789."},{"key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-319-95579-7","volume-title":"Studies in Conversational UX Design","author":"Moore Robert J.","year":"2018","unstructured":"Robert J. Moore and Raphael Arar. 2018. Conversational UX design: An introduction. In Studies in Conversational UX Design. Springer, 1\u201316."},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","DOI":"10.1145\/3304087","volume-title":"Conversational UX Design: A Practitioner\u2019s Guide to the Natural Conversation Framework","author":"Moore Robert J.","year":"2019","unstructured":"Robert J. Moore and Raphael Arar. 2019. Conversational UX Design: A Practitioner\u2019s Guide to the Natural Conversation Framework. ACM, New York, NY, USA."},{"key":"e_1_3_3_40_2","first-page":"45:1\u201345:4","volume-title":"Proc. 2nd Conf. on Conversational User Interfaces","author":"Moore Robert J.","year":"2020","unstructured":"Robert J. Moore, Eric Young Liu, Saurabh Mishra, and Guang-Jie Ren. 2020. Design systems for conversational UX. In Proc. 2nd Conf. on Conversational User Interfaces. ACM, 45:1\u201345:4."},{"issue":"5","key":"e_1_3_3_41_2","first-page":"91:1\u201391:42","article-title":"Software-based dialogue systems: Survey, taxonomy and challenges","volume":"55","author":"Motger Quim","year":"2023","unstructured":"Quim Motger, Xavier Franch, and Jordi Marco. 2023. Software-based dialogue systems: Survey, taxonomy and challenges. ACM Comput. Surv. 55, 5 (2023), 91:1\u201391:42.","journal-title":"ACM Comput. Surv."},{"issue":"6","key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.1007\/s10270-021-00913-x","article-title":"Evaluation of a machine learning classifier for metamodels","volume":"20","author":"Nguyen Phuong Thanh","year":"2021","unstructured":"Phuong Thanh Nguyen, Juri Di Rocco, Ludovico Iovino, Davide Di Ruscio, and Alfonso Pierantonio. 2021. Evaluation of a machine learning classifier for metamodels. Softw. Syst. Model. 20, 6 (2021), 1797\u20131821.","journal-title":"Softw. Syst. Model."},{"key":"e_1_3_3_43_2","unstructured":"OpenAI. 2023. ChatGPT. https:\/\/openai.com\/chatgpt (2023)."},{"key":"e_1_3_3_44_2","unstructured":"Pandorabots. 2023. https:\/\/home.pandorabots.com\/ (2023)."},{"key":"e_1_3_3_45_2","first-page":"89","volume-title":"Proc. 33rd Int. Scientific Conf. on Economic and Social Development","author":"Peras Dijana","year":"2018","unstructured":"Dijana Peras. 2018. Chatbot evaluation metrics: Review paper. In Proc. 33rd Int. Scientific Conf. on Economic and Social Development. Varazdin Development and Entrepreneurship Agency, 89\u201397."},{"issue":"6","key":"e_1_3_3_46_2","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/MS.2018.290101511","article-title":"Collaborative modeling and group decision making using chatbots in social networks","volume":"35","author":"P\u00e9rez-Soler Sara","year":"2018","unstructured":"Sara P\u00e9rez-Soler, Esther Guerra, and Juan de Lara. 2018. Collaborative modeling and group decision making using chatbots in social networks. IEEE Softw. 35, 6 (2018), 48\u201354.","journal-title":"IEEE Softw."},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/978-3-030-62522-1_15","volume-title":"Proc. 39th Int. Conf. on Conceptual Modeling (LNCS)","author":"P\u00e9rez-Soler Sara","year":"2020","unstructured":"Sara P\u00e9rez-Soler, Esther Guerra, and Juan de Lara. 2020. Model-driven chatbot development. In Proc. 39th Int. Conf. on Conceptual Modeling (LNCS), Vol. 12400. Springer, 207\u2013222."},{"key":"e_1_3_3_48_2","first-page":"37","volume-title":"Proc. 43rd IEEE\/ACM Int. Conf. on Soft. Eng.: Companion Proceedings","author":"P\u00e9rez-Soler Sara","year":"2021","unstructured":"Sara P\u00e9rez-Soler, Esther Guerra, and Juan de Lara. 2021. Creating and migrating chatbots with CONGA. In Proc. 43rd IEEE\/ACM Int. Conf. on Soft. Eng.: Companion Proceedings. IEEE, 37\u201340."},{"issue":"4","key":"e_1_3_3_49_2","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1109\/MS.2020.3030198","article-title":"Choosing a chatbot development tool","volume":"38","author":"P\u00e9rez-Soler Sara","year":"2021","unstructured":"Sara P\u00e9rez-Soler, Sandra Ju\u00e1rez-Puerta, Esther Guerra, and Juan de Lara. 2021. Choosing a chatbot development tool. IEEE Softw. 38, 4 (2021), 94\u2013103.","journal-title":"IEEE Softw."},{"key":"e_1_3_3_50_2","first-page":"186","volume-title":"Proc. Conf. on Empirical Methods in Natural Language Processing","author":"Pitler Emily","year":"2008","unstructured":"Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proc. Conf. on Empirical Methods in Natural Language Processing. ACL, USA, 186\u2013195."},{"key":"e_1_3_3_51_2","volume-title":"The English (Porter2) Stemming Algorithm","author":"Porter Martin","year":"2001","unstructured":"Martin Porter and Richard Boulton. 2001. The English (Porter2) Stemming Algorithm. http:\/\/snowball.tartarus.org\/algorithms\/english\/stemmer.html"},{"key":"e_1_3_3_52_2","first-page":"21","article-title":"Evaluating Quality of Chatbots and Intelligent Conversational Agents","volume":"1704","author":"Radziwill Nicole M.","year":"2017","unstructured":"Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. CoRR abs\/1704.04579 (2017), 21. http:\/\/arxiv.org\/abs\/1704.04579","journal-title":"CoRR"},{"key":"e_1_3_3_53_2","unstructured":"Rasa. 2023. https:\/\/rasa.com\/ (2023)."},{"issue":"11","key":"e_1_3_3_54_2","first-page":"1673","article-title":"Evaluation techniques for chatbot usability: A systematic mapping study","volume":"29","author":"Ren Ranci","year":"2019","unstructured":"Ranci Ren, John W. Castro, Silvia Teresita Acu\u00f1a, and Juan de Lara. 2019. Evaluation techniques for chatbot usability: A systematic mapping study. Int. J. Softw. Eng. Knowl. Eng. 29, 11&12 (2019), 1673\u20131702.","journal-title":"Int. J. Softw. Eng. Knowl. Eng."},{"issue":"1","key":"e_1_3_3_55_2","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1109\/TSE.2022.3150720","article-title":"Using the SOCIO chatbot for UML modelling: A family of experiments","volume":"49","author":"Ren Ranci","year":"2023","unstructured":"Ranci Ren, John W. Castro, Adri\u00e1n Santos, Oscar Dieste, and Silvia Teresita Acu\u00f1a. 2023. Using the SOCIO chatbot for UML modelling: A family of experiments. IEEE Trans. Software Eng. 49, 1 (2023), 364\u2013383.","journal-title":"IEEE Trans. Software Eng."},{"key":"e_1_3_3_56_2","first-page":"269","volume-title":"A Survey of Clustering Algorithms","author":"Rokach Lior","year":"2010","unstructured":"Lior Rokach. 2010. A Survey of Clustering Algorithms. Springer US, Boston, MA, 269\u2013298."},{"key":"e_1_3_3_57_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw Peter J.","year":"1987","unstructured":"Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. of Computational and Applied Mathematics 20 (1987), 53\u201365.","journal-title":"J. of Computational and Applied Mathematics"},{"key":"e_1_3_3_58_2","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1007\/978-0-387-30164-8","volume-title":"Encyclopedia of Machine Learning","author":"Sammut Claude","year":"2010","unstructured":"Claude Sammut and Geoffrey I. Webb. 2010. TF\u2013IDF. In Encyclopedia of Machine Learning. Springer US, Boston, MA, 986\u2013987."},{"issue":"6","key":"e_1_3_3_59_2","doi-asserted-by":"crossref","first-page":"1905","DOI":"10.1109\/TSE.2020.3042553","article-title":"Software module clustering: An in-depth literature analysis","volume":"48","author":"Sarhan Qusay Idrees","year":"2022","unstructured":"Qusay Idrees Sarhan, Bestoun S. Ahmed, Miroslav Bures, and Kamal Z. Zamli. 2022. Software module clustering: An in-depth literature analysis. IEEE Trans. Software Eng. 48, 6 (2022), 1905\u20131928.","journal-title":"IEEE Trans. Software Eng."},{"key":"e_1_3_3_60_2","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511791208","volume-title":"Sequence Organization in Interaction","author":"Schegloff Emanuel A.","year":"2007","unstructured":"Emanuel A. Schegloff. 2007. Sequence Organization in Interaction. Cambridge University Press."},{"key":"e_1_3_3_61_2","first-page":"60","volume-title":"Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics (Demonstrations)","author":"Sedoc Jo\u00e3o","year":"2019","unstructured":"Jo\u00e3o Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. ChatEval: A tool for chatbot evaluation. In Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics (Demonstrations). ACL, 60\u201365."},{"key":"e_1_3_3_62_2","volume-title":"Designing Bots: Creating Conversational Experiences","author":"Shevat Amir","year":"2017","unstructured":"Amir Shevat. 2017. Designing Bots: Creating Conversational Experiences. O\u2019Reilly."},{"key":"e_1_3_3_63_2","first-page":"792024:1\u2013792024","article-title":"Clustering methodologies for software engineering","volume":"2012","author":"Shtern Mark","year":"2012","unstructured":"Mark Shtern and Vassilios Tzerpos. 2012. Clustering methodologies for software engineering. Adv. Softw. Eng. 2012 (2012), 792024:1\u2013792024:18.","journal-title":"Adv. Softw. Eng."},{"key":"e_1_3_3_64_2","first-page":"1631","volume-title":"Proc. 2013 Conference on Empirical Methods in Natural Language Processing","author":"Socher Richard","year":"2013","unstructured":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1631\u20131642."},{"key":"e_1_3_3_65_2","volume-title":"EMF: Eclipse Modeling Framework, 2nd Edition","author":"Steinberg Dave","year":"2008","unstructured":"Dave Steinberg, Frank Budinsky, Ed Merks, and Marcelo Paternostro. 2008. EMF: Eclipse Modeling Framework, 2nd Edition. Pearson Education."},{"key":"e_1_3_3_66_2","first-page":"271","volume-title":"Proc. 35th Annual Meeting of the Association for Computational Linguistics and 8th Conf. of the Eur. Chapter of the Association for Computational Linguistics","author":"Walker Marilyn A.","year":"1997","unstructured":"Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proc. 35th Annual Meeting of the Association for Computational Linguistics and 8th Conf. of the Eur. Chapter of the Association for Computational Linguistics. Morgan Kaufmann Publishers \/ ACL, 271\u2013280."},{"key":"e_1_3_3_67_2","unstructured":"Watson. 2023. https:\/\/www.ibm.com\/cloud\/watson-assistant\/ (2023)."},{"issue":"1","key":"e_1_3_3_68_2","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1145\/365153.365168","article-title":"ELIZA - A computer program for the study of natural language communication between man and machine","volume":"9","author":"Weizenbaum Joseph","year":"1966","unstructured":"Joseph Weizenbaum. 1966. ELIZA - A computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36\u201345.","journal-title":"Commun. ACM"},{"key":"e_1_3_3_69_2","first-page":"60","volume-title":"Proc. 2019 IEEE Int. Conf. on Humanized Computing and Communication","author":"Xu Luxun","year":"2019","unstructured":"Luxun Xu, Vagelis Hristidis, and Nhat X. T. Le. 2019. Clustering-based summarization of transactional chatbot logs. In Proc. 2019 IEEE Int. Conf. on Humanized Computing and Communication. IEEE, 60\u201367."},{"key":"e_1_3_3_70_2","doi-asserted-by":"crossref","first-page":"2206","DOI":"10.1145\/3563657.3596138","volume-title":"Proceedings of the 2023 ACM Designing Interactive Systems Conference","author":"Zamfirescu-Pereira J. D.","year":"2023","unstructured":"J. D. Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G. Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI cats: Lessons from designing a chatbot by prompting GPT-3. In Proceedings of the 2023 ACM Designing Interactive Systems Conference, Daragh Byrne, Nikolas Martelaro, Andy Boucher, David J. Chatting, Sarah Fdili Alaoui, Sarah E. Fox, Iohanna Nicenboim, and Cayley MacArthur (Eds.). ACM, 2206\u20132220."},{"key":"e_1_3_3_71_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong Yifan Du Chen Yang Yushuo Chen Zhipeng Chen Jinhao Jiang Ruiyang Ren Yifan Li Xinyu Tang Zikang Liu Peiyu Liu Jian-Yun Nie and Ji-Rong Wen. 2023. A Survey of Large Language Models. (2023). arXiv preprint arXiv:1803.11175."},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2004.1274907"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637228","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637228","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:43:42Z","timestamp":1750290222000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637228"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":71,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3637228"],"URL":"https:\/\/doi.org\/10.1145\/3637228","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2023-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-29","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}