{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:13:03Z","timestamp":1750219983856,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,3,25]],"date-time":"2023-03-25T00:00:00Z","timestamp":1679702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100008982","name":"Qatar National Research Fund","doi-asserted-by":"crossref","award":["NPRP13S-0112-200037"],"award-info":[{"award-number":["NPRP13S-0112-200037"]}],"id":[{"id":"10.13039\/100008982","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>Learning response generation models constitute the main component of building open-domain dialogue systems. However, training open-domain response generation models requires large amounts of labeled data and pre-trained language generation models that are often nonexistent for low-resource languages. In this article, we propose a framework for training open-domain response generation models in low-resource settings. We consider Dialectal Arabic (DA) as a working example. The framework starts by warm-starting a transformer-based encoder-decoder with pre-trained language model parameters. Next, the resultant encoder-decoder model is adapted to DA by employing self-supervised pre-training on large-scale unlabeled data in the desired dialect. Finally, the model is fine-tuned on a very small labeled dataset for open-domain response generation. The results show significant performance improvements on three spoken Arabic dialects after adopting the framework\u2019s three stages, highlighted by higher BLEU and lower Perplexity scores compared with multiple baseline models. Specifically, our models are capable of generating fluent responses in multiple dialects with an average human-evaluated fluency score above 4. Our data is made publicly available.<\/jats:p>","DOI":"10.1145\/3579164","type":"journal-article","created":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T13:37:24Z","timestamp":1672925844000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Open-Domain Response Generation in Low-Resource Settings using Self-Supervised Pre-Training of Warm-Started Transformers"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0049-9318","authenticated-orcid":false,"given":"Tarek","family":"Naous","sequence":"first","affiliation":[{"name":"American University of Beirut, Qatar University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5887-0023","authenticated-orcid":false,"given":"Zahraa","family":"Bassyouni","sequence":"additional","affiliation":[{"name":"American University of Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6668-9662","authenticated-orcid":false,"given":"Bassel","family":"Mousi","sequence":"additional","affiliation":[{"name":"American University of Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9954-7924","authenticated-orcid":false,"given":"Hazem","family":"Hajj","sequence":"additional","affiliation":[{"name":"American University of Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5206-2954","authenticated-orcid":false,"given":"Wassim El","family":"Hajj","sequence":"additional","affiliation":[{"name":"American University of Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5688-7515","authenticated-orcid":false,"given":"Khaled","family":"Shaban","sequence":"additional","affiliation":[{"name":"Qatar University, Qatar"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,3,25]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-3003"},{"key":"e_1_3_2_3_2","article-title":"ARBERT & MARBERT: Deep bidirectional transformers for Arabic","author":"Abdul-Mageed Muhammad","year":"2020","unstructured":"Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785 (2020).","journal-title":"arXiv preprint arXiv:2101.01785"},{"key":"e_1_3_2_4_2","first-page":"97","volume-title":"Proceedings of the 5th Arabic Natural Language Processing Workshop","author":"Abdul-Mageed Muhammad","year":"2020","unstructured":"Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the 5th Arabic Natural Language Processing Workshop. 97\u2013110."},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"5855","DOI":"10.18653\/v1\/2020.emnlp-main.472","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Abdul-Mageed Muhammad","year":"2020","unstructured":"Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020. Beyond geolocation: Micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5855\u20135876."},{"key":"e_1_3_2_6_2","article-title":"Towards a human-like open-domain chatbot","author":"Adiwardana Daniel","year":"2020","unstructured":"Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 (2020).","journal-title":"arXiv preprint arXiv:2001.09977"},{"key":"e_1_3_2_7_2","first-page":"208","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations","author":"Ali Dana Abu","year":"2016","unstructured":"Dana Abu Ali and Nizar Habash. 2016. Botta: An Arabic dialect chatbot. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations. 208\u2013212."},{"key":"e_1_3_2_8_2","first-page":"9","volume-title":"Proceedings of the LREC 2020 Workshop Language Resources and Evaluation Conference (11\u201316 May 2020)","author":"Antoun Wissam","year":"2020","unstructured":"Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the LREC 2020 Workshop Language Resources and Evaluation Conference (11\u201316 May 2020). 9."},{"key":"e_1_3_2_9_2","volume-title":"Proceedings of the 3rd International Conference on Learning Representations, (ICLR 2015)","author":"Bahdanau Dzmitry","year":"2015","unstructured":"Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, (ICLR 2015)."},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Basu Sourya","year":"2020","unstructured":"Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. 2020. Mirostat: A neural text decoding algorithm that directly controls perplexity. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.18653\/v1\/W19-4622","volume-title":"Proceedings of the 4th Arabic Natural Language Processing Workshop","author":"Bouamor Houda","year":"2019","unstructured":"Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the 4th Arabic Natural Language Processing Workshop. 199\u2013207."},{"key":"e_1_3_2_12_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Dinan Emily","year":"2018","unstructured":"Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_13_2","first-page":"585","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Eskander Ramy","year":"2013","unstructured":"Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013. Processing spontaneous orthography. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 585\u2013595."},{"key":"e_1_3_2_14_2","first-page":"295","volume-title":"Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)","author":"Fadhil Ahmed","year":"2019","unstructured":"Ahmed Fadhil and Ahmed AbuRaed. 2019. OlloBot-towards a text-based Arabic health conversational agent: Evaluation and results. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). 295\u2013303."},{"key":"e_1_3_2_15_2","article-title":"A survey on recent approaches for natural language processing in low-resource scenarios","author":"Hedderich Michael A.","year":"2020","unstructured":"Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str\u00f6tgen, and Dietrich Klakow. 2020. A survey on recent approaches for natural language processing in low-resource scenarios. arXiv preprint arXiv:2010.12309 (2020).","journal-title":"arXiv preprint arXiv:2010.12309"},{"key":"e_1_3_2_16_2","first-page":"928","volume-title":"Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers","author":"Higashinaka Ryuichiro","year":"2014","unstructured":"Ryuichiro Higashinaka, Kenji Imamura, Toyomi Meguro, Chiaki Miyazaki, Nozomi Kobayashi, Hiroaki Sugiyama, Toru Hirano, Toshiro Makino, and Yoshihiro Matsuo. 2014. Towards an open-domain conversational system fully based on natural language processing. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. 928\u2013939."},{"issue":"3","key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3383123","article-title":"Challenges in building intelligent open-domain dialog systems","volume":"38","author":"Huang Minlie","year":"2020","unstructured":"Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS) 38, 3 (2020), 1\u201332.","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"volume-title":"Proceedings of the 57th Conference of the Association for Computational Linguistics","year":"2018","key":"e_1_3_2_18_2","unstructured":"Daphne Ippolito, Reno Kriz, Maria Kustikova, Jo\u00e3o Sedoc, and Chris Callison-Burch. 2018. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Conference of the Association for Computational Linguistics. Association for Computational Linguistics."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_2_20_2","first-page":"986","volume-title":"Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Li Yanran","year":"2017","unstructured":"Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 986\u2013995."},{"key":"e_1_3_2_21_2","first-page":"13622","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"34","author":"Lin Zhaojiang","year":"2020","unstructured":"Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020. Caire: An end-to-end empathetic chatbot. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 13622\u201313623."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491065"},{"issue":"2","key":"e_1_3_2_23_2","first-page":"1","article-title":"Improving NER tagging performance in low-resource languages via multilingual learning","volume":"18","author":"Murthy Rudra","year":"2018","unstructured":"Rudra Murthy, Mitesh M. Khapra, and Pushpak Bhattacharyya. 2018. Improving NER tagging performance in low-resource languages via multilingual learning. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 18, 2 (2018), 1\u201320.","journal-title":"ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)"},{"key":"e_1_3_2_24_2","first-page":"164","volume-title":"Proceedings of the 6th Arabic Natural Language Processing Workshop","author":"Naous Tarek","year":"2021","unstructured":"Tarek Naous, Wissam Antoun, Reem Mahmoud, and Hazem Hajj. 2021. Empathetic BERT2BERT conversational model: Learning Arabic language generation with little data. In Proceedings of the 6th Arabic Natural Language Processing Workshop. 164\u2013172."},{"key":"e_1_3_2_25_2","first-page":"58","volume-title":"Proceedings of the 5th Arabic Natural Language Processing Workshop","author":"Naous Tarek","year":"2020","unstructured":"Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020. Empathy-driven Arabic conversational chatbot. In Proceedings of the 5th Arabic Natural Language Processing Workshop. 58\u201368."},{"key":"e_1_3_2_26_2","first-page":"436","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Otegi Arantxa","year":"2020","unstructured":"Arantxa Otegi, Aitor Agirre, Jon Ander Campos, Aitor Soroa, and Eneko Agirre. 2020. Conversational question answering in low resource scenarios: A dataset and case study for Basque. In Proceedings of the 12th Language Resources and Evaluation Conference. 436\u2013442."},{"issue":"1","key":"e_1_3_2_27_2","first-page":"1","article-title":"Multilingual offensive language identification for low-resource languages","volume":"21","author":"Ranasinghe Tharindu","year":"2021","unstructured":"Tharindu Ranasinghe and Marcos Zampieri. 2021. Multilingual offensive language identification for low-resource languages. Transactions on Asian and Low-Resource Language Information Processing 21, 1 (2021), 1\u201313.","journal-title":"Transactions on Asian and Low-Resource Language Information Processing"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1534"},{"key":"e_1_3_2_29_2","article-title":"Open-domain conversational agents: Current progress, open problems, and future directions","author":"Roller Stephen","year":"2020","unstructured":"Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, et\u00a0al. 2020. Open-domain conversational agents: Current progress, open problems, and future directions. arXiv preprint arXiv:2006.12442 (2020).","journal-title":"arXiv preprint arXiv:2006.12442"},{"key":"e_1_3_2_30_2","first-page":"300","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Roller Stephen","year":"2021","unstructured":"Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 300\u2013325."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00313"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700826"},{"key":"e_1_3_2_33_2","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 5998\u20136008.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457571"},{"key":"e_1_3_2_36_2","first-page":"483","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Xue Linting","year":"2021","unstructured":"Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 483\u2013498."},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"1886","DOI":"10.18653\/v1\/D19-1197","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","year":"2019","unstructured":"Ze Yang, Wei Wu, Jian Yang, Can Xu, and Zhoujun Li. 2019. Low-resource response generation with template prior. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 1886\u20131897."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1205"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-demos.30"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","first-page":"6556","DOI":"10.18653\/v1\/2020.emnlp-main.531","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Zhong Peixiang","year":"2020","unstructured":"Peixiang Zhong, Chen Zhang, Hao Wang, Yong Liu, and Chunyan Miao. 2020. Towards persona-based empathetic conversational models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6556\u20136566."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00368"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579164","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3579164","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:27Z","timestamp":1750182567000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579164"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,25]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3579164"],"URL":"https:\/\/doi.org\/10.1145\/3579164","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"type":"print","value":"2375-4699"},{"type":"electronic","value":"2375-4702"}],"subject":[],"published":{"date-parts":[[2023,3,25]]},"assertion":[{"value":"2022-03-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-28","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}