{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T03:51:59Z","timestamp":1784865119302,"version":"3.55.0"},"reference-count":42,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T00:00:00Z","timestamp":1674518400000},"content-version":"vor","delay-in-days":23,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,1,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose RAG-end2end, an extension to RAG that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. This auxiliary signal forces RAG-end2end to reconstruct a given sentence by accessing the relevant information from the external knowledge base. Our novel contribution is that, unlike RAG, RAG-end2end does joint training of the retriever and generator for the end QA task and domain adaptation. We evaluate our approach with datasets from three domains: COVID-19, News, and Conversations, and achieve significant performance improvements compared to the original RAG model. Our work has been open-sourced through the HuggingFace Transformers library, attesting to our work\u2019s credibility and technical consistency.<\/jats:p>","DOI":"10.1162\/tacl_a_00530","type":"journal-article","created":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T16:17:15Z","timestamp":1674577035000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":290,"title":["Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering"],"prefix":"10.1162","volume":"11","author":[{"given":"Shamane","family":"Siriwardhana","sequence":"first","affiliation":[{"name":"Augmented Human Lab, Auckland Bioengineering Institute, The University of Auckland, New Zealand. Shamane@ahlab.org"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rivindu","family":"Weerasekera","sequence":"additional","affiliation":[{"name":"Augmented Human Lab, Auckland Bioengineering Institute, The University of Auckland, New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elliott","family":"Wen","sequence":"additional","affiliation":[{"name":"Augmented Human Lab, Auckland Bioengineering Institute, The University of Auckland, New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tharindu","family":"Kaluarachchi","sequence":"additional","affiliation":[{"name":"Augmented Human Lab, Auckland Bioengineering Institute, The University of Auckland, New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajib","family":"Rana","sequence":"additional","affiliation":[{"name":"University of Southern Queensland, Australia. Rajib.Rana@usq.edu.au"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suranga","family":"Nanayakkara","sequence":"additional","affiliation":[{"name":"Department of Information Systems & Analytics, National University of Singapore, Singapore"},{"name":"Augmented Human Lab, Auckland Bioengineering Institute, The University of Auckland, New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"key":"2023012416102754200_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1620","article-title":"Synthetic QA corpora generation with roundtrip consistency","author":"Alberti","year":"2019","journal-title":"arXiv preprint arXiv:1906.05416"},{"key":"2023012416102754200_bib2","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1007\/978-3-319-24027-5_20","article-title":"Modeling of the question answering task in the yodaQA system","volume-title":"International Conference of the Cross-language Evaluation Forum for European Languages","author":"Baudi\u0161","year":"2015"},{"key":"2023012416102754200_bib3","first-page":"1533","article-title":"Semantic parsing on freebase from question-answer pairs","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Berant","year":"2013"},{"key":"2023012416102754200_bib4","article-title":"Improving language models by retrieving from trillions of tokens","author":"Borgeaud","year":"2021","journal-title":"arXiv preprint arXiv:2112.04426"},{"key":"2023012416102754200_bib5","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023012416102754200_bib6","doi-asserted-by":"publisher","first-page":"3340","DOI":"10.18653\/v1\/2022.acl-long.236","article-title":"Hallucinated but factual! Inspecting the factuality of hallucinations in abstractive summarization","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Cao","year":"2022"},{"key":"2023012416102754200_bib7","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv preprint arXiv:1810.04805"},{"key":"2023012416102754200_bib8","article-title":"REALM: Retrieval-augmented language model pre-training","author":"Guu","year":"2020","journal-title":"arXiv preprint arXiv:2002.08909"},{"key":"2023012416102754200_bib9","first-page":"1693","article-title":"Teaching machines to read and comprehend","volume":"28","author":"Hermann","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023012416102754200_bib10","article-title":"Billion-scale similarity search with GPUs","author":"Johnson","year":"2017","journal-title":"arXiv preprint arXiv:1702.08734"},{"key":"2023012416102754200_bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147","article-title":"TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension","author":"Joshi","year":"2017","journal-title":"arXiv preprint arXiv:1705.03551"},{"key":"2023012416102754200_bib12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","author":"Karpukhin","year":"2020","journal-title":"arXiv preprint arXiv:2004.04906"},{"key":"2023012416102754200_bib13","article-title":"Ctrl: A conditional transformer language model for controllable generation","author":"Keskar","year":"2019","journal-title":"arXiv preprint arXiv:1909.05858"},{"key":"2023012416102754200_bib14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.579","article-title":"Internet-augmented dialogue generation","author":"Komeili","year":"2021","journal-title":"arXiv preprint arXiv:2107.07566"},{"key":"2023012416102754200_bib15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.750","article-title":"Evaluating the factual consistency of abstractive text summarization","author":"Kry\u015bci\u0144ski","year":"2019","journal-title":"arXiv preprint arXiv:1910.12840"},{"key":"2023012416102754200_bib16","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023012416102754200_bib17","article-title":"Latent retrieval for weakly supervised open domain question answering","author":"Lee","year":"2019","journal-title":"arXiv preprint arXiv:1906.00300"},{"key":"2023012416102754200_bib18","article-title":"Pre-training via paraphrasing","author":"Lewis","year":"2020","journal-title":"arXiv preprint arXiv:2006.15020"},{"key":"2023012416102754200_bib19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019","journal-title":"arXiv preprint arXiv:1910.13461"},{"key":"2023012416102754200_bib20","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","author":"Lewis","year":"2020","journal-title":"arXiv preprint arXiv:2005.11401"},{"key":"2023012416102754200_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.86","article-title":"Question and answer test-train overlap in open-domain question answering datasets","author":"Lewis","year":"2020","journal-title":"arXiv preprint arXiv:2008.02637"},{"key":"2023012416102754200_bib22","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00415","article-title":"Paq: 65 million probably-asked questions and what you can do with them","author":"Lewis","year":"2021","journal-title":"arXiv preprint arXiv:2102.07033"},{"key":"2023012416102754200_bib23","doi-asserted-by":"publisher","DOI":"10.3115\/1118108.1118117","article-title":"NLTK: The natural language toolkit","author":"Loper","year":"2002","journal-title":"arXiv preprint cs\/0205028"},{"key":"2023012416102754200_bib24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.92","article-title":"Zero-shot neural retrieval via domain-targeted synthetic query generation","author":"Ji","year":"2020","journal-title":"arXiv preprint arXiv:2004.14503"},{"key":"2023012416102754200_bib25","article-title":"NeurIPS 2020 EfficientQA competition: Systems, analyses and lessons learned","author":"Min","year":"2021","journal-title":"arXiv preprint arXiv:2101.00133v1"},{"key":"2023012416102754200_bib26","article-title":"COVID-QA: A question answering dataset for COVID-19","volume-title":"Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020","author":"Moller","year":"2020"},{"key":"2023012416102754200_bib27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458817.3476209","article-title":"Efficient large-scale language model training on GPU clusters using Megatron-LM","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","author":"Narayanan","year":"2021"},{"key":"2023012416102754200_bib28","doi-asserted-by":"publisher","first-page":"2673","DOI":"10.18653\/v1\/P19-1256","article-title":"A simple recipe towards reducing hallucination in neural surface realisation","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Nie","year":"2019"},{"key":"2023012416102754200_bib29","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019","journal-title":"arXiv preprint arXiv:1910.10683"},{"key":"2023012416102754200_bib30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264","article-title":"SQUAD: 100,000+ questions for machine comprehension of text","author":"Rajpurkar","year":"2016","journal-title":"arXiv preprint arXiv:1606.05250"},{"key":"2023012416102754200_bib31","doi-asserted-by":"publisher","DOI":"10.1561\/1500000019","volume-title":"The Probabilistic Relevance Framework: BM25 and Beyond","author":"Robertson","year":"2009"},{"key":"2023012416102754200_bib32","article-title":"End-to-end training of neural retrievers for open-domain question answering","author":"Sachan","year":"2021","journal-title":"arXiv preprint arXiv:2101.00408"},{"key":"2023012416102754200_bib33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.439","article-title":"End-to-end synthetic data generation for domain adaptation of question answering systems","author":"Shakeri","year":"2020","journal-title":"arXiv preprint arXiv:2010.06028"},{"key":"2023012416102754200_bib34","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.320","article-title":"Retrieval augmentation reduces hallucination in conversation","author":"Shuster","year":"2021","journal-title":"arXiv preprint arXiv:2104.07567"},{"key":"2023012416102754200_bib35","article-title":"End-to-end training of multi-document reader and retriever for open-domain question answering","volume":"34","author":"Singh","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023012416102754200_bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-2623","article-title":"NewsQA: A machine comprehension dataset","author":"Trischler","year":"2016","journal-title":"arXiv preprint arXiv:1611.09830"},{"key":"2023012416102754200_bib37","article-title":"CORD-19: The COVID-19 open research dataset","author":"Wang","year":"2020","journal-title":"arXiv preprint arXiv:2004.10706v2"},{"key":"2023012416102754200_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"HuggingFace\u2019s transformers: State-of-the-art natural language processing","author":"Wolf","year":"2019","journal-title":"arXiv preprint arXiv:1910.03771"},{"key":"2023012416102754200_bib39","article-title":"Controllable abstractive dialogue summarization with sketch supervision","author":"Chien-Sheng","year":"2021","journal-title":"arXiv preprint arXiv:2105.14064"},{"key":"2023012416102754200_bib40","article-title":"QAconv: Question answering on informative conversations","author":"Chien-Sheng","year":"2021","journal-title":"arXiv preprint arXiv:2105.06912"},{"key":"2023012416102754200_bib41","article-title":"Beyond goldfish memory: Long-term open-domain conversation","author":"Jing","year":"2021","journal-title":"arXiv preprint arXiv: 2107.07567"},{"key":"2023012416102754200_bib42","doi-asserted-by":"publisher","first-page":"2013","DOI":"10.18653\/v1\/D15-1237","article-title":"WikiQA: A challenge dataset for open-domain question answering","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Yi","year":"2015"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00530\/2067834\/tacl_a_00530.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00530\/2067834\/tacl_a_00530.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,24]],"date-time":"2023-01-24T16:17:29Z","timestamp":1674577049000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00530\/114590\/Improving-the-Domain-Adaptation-of-Retrieval"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":42,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00530","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}