{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T10:14:12Z","timestamp":1791972852995,"version":"4.3.4"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>Software chatbots are growing in popularity and have been increasingly used in software projects due to their benefits in saving time, cost, and effort. At the core of every chatbot is a Natural Language Understanding (NLU) component that enables chatbots to comprehend the users\u2019 queries. Prior work shows that chatbot practitioners face challenges in training the NLUs because the labeled training data is scarce. Consequently, practitioners resort to user queries to enhance chatbot performance. They annotate these queries and use them for NLU training. However, such training is done manually and prohibitively expensive. Therefore, we propose AlphaBot to automate the query annotation process for SE chatbots. Specifically, we leverage weak supervision to label users\u2019 queries posted to a software repository-based chatbot. To evaluate the impact of using AlphaBot on the NLU\u2019s performance, we conducted a case study using a dataset that comprises 749 queries and 52 intents. The results show that using AlphaBot improves the NLU\u2019s performance in terms of F1-score, with improvements ranging from 0.96% to 35%. Furthermore, our results show that applying more labeling functions improves the NLU\u2019s classification of users\u2019 queries. Our work enables practitioners to focus on their chatbots\u2019 core functionalities rather than annotating users\u2019 queries.<\/jats:p>","DOI":"10.1145\/3660812","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"2378-2401","source":"Crossref","is-referenced-by-count":3,"title":["A Weak Supervision-Based Approach to Improve Chatbots for Code Repositories"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1637-9898","authenticated-orcid":false,"given":"Farbod","family":"Farhour","sequence":"first","affiliation":[{"name":"Concordia University, Montreal, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1863-9147","authenticated-orcid":false,"given":"Ahmad","family":"Abdellatif","sequence":"additional","affiliation":[{"name":"University of Calgary, Calgary, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6851-6351","authenticated-orcid":false,"given":"Essam","family":"Mansour","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1285-9878","authenticated-orcid":false,"given":"Emad","family":"Shihab","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3078384"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09788-5"},{"key":"e_1_3_1_4_2","unstructured":"Ahmad Abdellatif Khaled Badran and Emad Shihab. 2021. AskGit. https:\/\/askgit.io\/ (Accessed on 12\/07\/2023)."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387472"},{"key":"e_1_3_1_6_2","unstructured":"Enrique Alfonseca Katja Filippova Jean-Yves Delort and Guillermo Garrido. 2012. Pattern Learning for Relation Extraction with a Hierarchical Topic Model. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 (Jeju Island Korea) (ACL \u201812). Association for Computational Linguistics USA 54\u201359."},{"issue":"118","key":"e_1_3_1_7_2","first-page":"1","article-title":"A General Framework for Adversarial Label Learning","volume":"22","author":"Arachie Chidubem","year":"2021","unstructured":"Chidubem Arachie and Bert Huang. 2021. A General Framework for Adversarial Label Learning. Journal of Machine Learning Research 22, 118 (2021), 1\u201333.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_8_2","article-title":"The State Of Chatbots","volume":"20","author":"Ask Julie","year":"2016","unstructured":"Julie Ask, Michael Facemire, and Andrew Hogan. 2016. The State Of Chatbots. Forrester.com report 20 (2016).","journal-title":"Forrester.com report"},{"key":"e_1_3_1_9_2","first-page":"273","volume-title":"International Conference on Machine Learning","author":"Bach Stephen H","year":"2017","unstructured":"Stephen H Bach, Bryan He, Alexander Ratner, and Christopher Re. 2017. Learning the structure of generative models without labeled data. In International Conference on Machine Learning. PMLR, 273\u2013282."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314036"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.03.159"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCES48766.2020.9137944"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180238"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5522"},{"key":"e_1_3_1_15_2","unstructured":"Metz C. 2016. Google\u2019s Hand-Fed AI Now Gives Answers Not Just Search Results. https:\/\/www.wired.com\/2016\/11\/googles-search-engine-can-now-answer-questions-human-help\/ (Accessed on 08\/02\/2023)."},{"key":"e_1_3_1_16_2","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1007\/978-3-030-01689-0_23","volume-title":"Cyberspace Safety and Security","author":"Clarizia Fabio","year":"2018","unstructured":"Fabio Clarizia, Francesco Colace, Marco Lombardi, Francesco Pascale, and Domenico Santaniello. 2018. Chatbot: An Education Support System for Student. In Cyberspace Safety and Security, Arcangelo Castiglione, Florin Pop, Massimo Ficco, and Francesco Palmieri (Eds.). Springer International Publishing, Cham, 291\u2013302."},{"key":"e_1_3_1_17_2","article-title":"A CTD-Pfizer collaboration: manual curation of 88 000 scientific articles text mined for drug-disease and drug-phenotype interactions","volume":"2013","author":"Davis Allan Peter","year":"2013","unstructured":"Allan Peter Davis, Thomas C Wiegers, Phoebe M Roberts, Benjamin L King, Jean M Lay, Kelley Lennon-Hopkins, Daniela Sciaky, Robin Johnson, Heather Keating, Nigel Greene, et al. 2013. A CTD-Pfizer collaboration: manual curation of 88 000 scientific articles text mined for drug-disease and drug-phenotype interactions. Database 2013 (2013).","journal-title":"Database"},{"key":"e_1_3_1_18_2","unstructured":"Dialogflow. 2021. Dialogflow Official Website. https:\/\/dialogflow.cloud.google.com\/ (Accessed on 01\/08\/2023)."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391534"},{"key":"e_1_3_1_20_2","unstructured":"Farbod Farhour Ahmad Abdellatif Essam Mansour and Emad Shihab. 2023. A Weak Supervision-based Approach to Improve Chatbots for Code Repositories. https:\/\/zenodo.org\/records\/10714394 (Accessed on 28\/09\/2023)."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-22328-4"},{"key":"e_1_3_1_22_2","first-page":"731","volume-title":"Incremental Learning","author":"Geng Xin","year":"2009","unstructured":"Xin Geng and Kate Smith-Miles. 2009. Incremental Learning. Springer US, Boston, MA, 731\u2013735."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICAEE.2017.8255404"},{"key":"e_1_3_1_24_2","unstructured":"GoodRebels. [n. d.]. The impact of conversational bots in the customer experience. https:\/\/www.goodrebels.com\/the-impact-of-conversational-bots-in-the--experience\/ (Accessed on 12\/07\/2023)."},{"key":"e_1_3_1_25_2","article-title":"Learning from dialogue after deployment: Feed yourself, chatbot!","author":"Hancock Braden","year":"2019","unstructured":"Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415 (2019).","journal-title":"arXiv preprint arXiv:1901.05415"},{"key":"e_1_3_1_26_2","article-title":"The Rise of Crowdsourcing","volume":"14","year":"2006","unstructured":"Howe and Jeff. 2006. The Rise of Crowdsourcing. Wired 14 (01 2006).","journal-title":"Wired"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556925"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2672677"},{"key":"e_1_3_1_29_2","unstructured":"jsphdnl. 2017. nlp - Conversational Data for building a chat bot - Stack Overflow. https:\/\/stackoverflow.com\/questions\/45821517\/conversational-data-for-building-a-chat-bot (Accessed on 08\/14\/2023)."},{"key":"e_1_3_1_30_2","unstructured":"Pensive Knave. 2021. reactjs - How to create CSS underline which partially covers the word? - Stack Overflow. https:\/\/stackoverflow.com\/questions\/67694697\/how-to-create-css-underline-which-partially-covers-the-word (Accessed on 08\/25\/2023)."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.4541027"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391501"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","first-page":"9528","DOI":"10.1609\/aaai.v33i01.33019528","article-title":"Bootstrapping conversational agents with weak supervision","volume":"33","author":"Mallinar Neil","year":"2019","unstructured":"Neil Mallinar, Abhishek Shah, Rajendra Ugrani, Ayush Gupta, Manikandan Gurusankar, Tin Kam Ho, Q Vera Liao, Yunfeng Zhang, Rachel KE Bellamy, Robert Yates, et al. 2019. Bootstrapping conversational agents with weak supervision. In Proceedings of the AAAIConference on Artificial Intelligence, Vol. 33. 9528\u20139533.","journal-title":"Proceedings of the AAAIConference on Artificial Intelligence"},{"issue":"32","key":"e_1_3_1_34_2","first-page":"955","article-title":"Generalized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data","volume":"11","author":"Mann Gideon S.","year":"2010","unstructured":"Gideon S. Mann and Andrew McCallum. 2010. Generalized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data. Journal of Machine Learning Research 11, 32 (2010), 955\u2013984.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271737"},{"key":"e_1_3_1_36_2","unstructured":"Microsoft. 2021. Language Understanding - Bot Service. https:\/\/docs.microsoft.com\/en-us\/azure\/bot-service\/bot-builder-concept-luis?view=azure-bot-service-4.0#best-practices-for-language-understanding (Accessed on 08\/12\/2023)."},{"key":"e_1_3_1_37_2","unstructured":"Microsoft. 2021. LUIS (Language Understanding) - Cognitive Services. https:\/\/www.luis.ai\/ (Accessed on 12\/09\/2023)."},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Mike Mintz Steven Bills Rion Snow and Dan Jurafsky. 2009. Distant Supervision for Relation Extraction without Labeled Data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2 (Suntec Singapore) (ACL \u201809). Association for Computational Linguistics USA 1003\u20131011.","DOI":"10.3115\/1690219.1690287"},{"key":"e_1_3_1_39_2","unstructured":"Eadicicco L. Baidu\u2019s Andrew Ng. 2017. on the future of artificial intelligence. https:\/\/time.com\/4631730\/andrew-ng-artificial-intelligence-2017\/ (Accessed on 07\/24\/2023)."},{"key":"e_1_3_1_40_2","first-page":"4","article-title":"AI-based chatbot service for financial industry","volume":"54","author":"Okuda T.","year":"2018","unstructured":"T. Okuda and S. Shoda. 2018. AI-based chatbot service for financial industry. Fujitsu Scientific and Technical Journal 54 (04 2018), 4\u20138.","journal-title":"Fujitsu Scientific and Technical Journal"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-industry.7"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE.2019.00016"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.335"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/5326.983933"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE52550.2021.00015"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00077"},{"key":"e_1_3_1_47_2","unstructured":"Rasa. [n. d.]. Duckling. https:\/\/duckling.wit.ai\/ (Accessed on 08\/14\/2023)."},{"key":"e_1_3_1_48_2","unstructured":"Rasa. 2021. Introduction to Rasa X. https:\/\/rasa.com\/docs\/rasa-x\/ (Accessed on 07\/29\/2023)."},{"key":"e_1_3_1_49_2","unstructured":"RASA. 2021. Open source conversational AI | Rasa. https:\/\/rasa.com\/ (Accessed on 12\/09\/2023)."},{"key":"e_1_3_1_50_2","first-page":"269","volume-title":"Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases","author":"Ratner Alexander","year":"2017","unstructured":"Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R\u00e9. 2017. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, Vol. 11. NIH Public Access, 269."},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00552-1"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2016.2636665"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358040"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391505"},{"key":"e_1_3_1_56_2","doi-asserted-by":"crossref","unstructured":"Massimo Ruffolo. and Francesco Visalli. 2020. A Weak-supervision Method for Automating Training Set Creation in Multi-domain Aspect Sentiment Classification. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART . INSTICC SciTePress 249\u2013256.","DOI":"10.5220\/0009165602490256"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSCAN.2019.8878779"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE52550.2021.00013"},{"key":"e_1_3_1_59_2","unstructured":"Sheri. 2020. python - Intent classification for Chatbot - Stack Overflow. https:\/\/stackoverflow.com\/questions\/62970861\/intent-classification-for-chatbot (Accessed on 09\/05\/2023)."},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6412"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401121"},{"key":"e_1_3_1_62_2","unstructured":"Spacy. 2021. Industrial-strength Natural Language Processing in Python. https:\/\/spacy.io\/ (Accessed on 01\/09\/2023)."},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2983989"},{"key":"e_1_3_1_64_2","unstructured":"Snorkel Team. 2020. An Overview of Weak Supervision. https:\/\/www.snorkel.org\/blog\/weak-supervision (Accessed on 07\/17\/2023)."},{"key":"e_1_3_1_65_2","unstructured":"Snorkel Team. 2020. Snorkel Intro Tutorial: Data Labeling. https:\/\/www.snorkel.org\/use-cases\/01-spam-tutorial (Accessed on 07\/09\/2023)."},{"key":"e_1_3_1_66_2","doi-asserted-by":"crossref","unstructured":"Carlos Toxtli Andr\u00e9s Monroy-Hernandez and Justin Cranshaw. 2018. Understanding Chatbot-Mediated Task Management. Association for Computing Machinery New York NY USA.","DOI":"10.1145\/3173574.3173632"},{"key":"e_1_3_1_67_2","unstructured":"Kieu Tran. 2020. nlp - Building a chatbot about literary novel - Stack Overflow. https:\/\/stackoverflow.com\/questions\/64007306\/building-a-chatbot-about-literary-novel (Accessed on 07\/16\/2023)."},{"key":"e_1_3_1_68_2","unstructured":"RASA. 2021. Incremental Training. https:\/\/rasa.com\/docs\/rasa\/next\/migration-guide#incremental-training (Accessed on 11\/08\/2023)."},{"issue":"1","key":"e_1_3_1_69_2","first-page":"7","article-title":"Preprocessing techniques for text mining-an overview","volume":"5","author":"Vijayarani S","year":"2015","unstructured":"S Vijayarani, Ms J Ilamathi, Ms Nithya, et al. 2015. Preprocessing techniques for text mining-an overview. International Journal of Computer Science & Communication Networks 5, 1 (2015), 7\u201316.","journal-title":"International Journal of Computer Science & Communication Networks"},{"key":"e_1_3_1_70_2","article-title":"Learning from Multiple Noisy Partial Labelers","author":"Yu Peilin","year":"2021","unstructured":"Peilin Yu, Tiffany Ding, and Stephen H Bach. 2021. Learning from Multiple Noisy Partial Labelers. arXiv preprint arXiv:2106.04530 (2021).","journal-title":"arXiv preprint arXiv:2106.04530"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/PASSAT\/SocialCom.2011.203"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3060586"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3016006"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1093\/nsr\/nwx106"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660812","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660812","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:02:28Z","timestamp":1770192148000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660812"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":73,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660812"],"URL":"https:\/\/doi.org\/10.1145\/3660812","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}