{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T05:53:59Z","timestamp":1695707639540},"reference-count":30,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,9,25]],"date-time":"2023-09-25T00:00:00Z","timestamp":1695600000000},"content-version":"vor","delay-in-days":267,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,9,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval, in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model\u2014one that is explicitly optimized for multitasking\u2014along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.1<\/jats:p>","DOI":"10.1162\/tacl_a_00597","type":"journal-article","created":{"date-parts":[[2023,9,25]],"date-time":"2023-09-25T19:50:16Z","timestamp":1695671416000},"page":"1201-1212","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving Multitask Retrieval by Promoting Task Specialization"],"prefix":"10.1162","volume":"11","author":[{"given":"Wenzheng","family":"Zhang","sequence":"first","affiliation":[{"name":"Rutgers University, USA. wenzheng.zhang@rutgers.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenyan","family":"Xiong","sequence":"additional","affiliation":[{"name":"Microsoft, USA. chenyan.xiong@microsoft.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karl","family":"Stratos","sequence":"additional","affiliation":[{"name":"Rutgers University, USA. karl.stratos@rutgers.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arnold","family":"Overwijk","sequence":"additional","affiliation":[{"name":"Microsoft, USA. arnold.overwijk@microsoft.com"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,9,25]]},"reference":[{"key":"2023092519501002300_bib1","article-title":"Task-aware retrieval with instructions","author":"Asai","year":"2022","journal-title":"arXiv preprint arXiv:2211.09260"},{"key":"2023092519501002300_bib2","article-title":"Autoregressive search engines: Generating substrings as document identifiers","author":"Bevilacqua","year":"2022","journal-title":"arXiv preprint arXiv:2204.10628"},{"key":"2023092519501002300_bib3","doi-asserted-by":"publisher","DOI":"10.1142\/S0218001493000339","article-title":"Signature verification using a \u201cSiamese\u201d time delay neural network","volume":"6","author":"Bromley","year":"1993","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023092519501002300_bib4","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557271","article-title":"Corpusbrain: Pre-train a generative retrieval model for knowledge-intensive language tasks","author":"Chen","year":"2022","journal-title":"arXiv preprint arXiv:2208.07652"},{"key":"2023092519501002300_bib5","first-page":"794","article-title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","volume-title":"International Conference on Machine Learning","author":"Chen","year":"2018"},{"key":"2023092519501002300_bib6","article-title":"Autoregressive entity retrieval","volume-title":"International Conference on Learning Representations","author":"De Cao","year":"2021"},{"key":"2023092519501002300_bib10","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of NAACL-HLT 2019","author":"Devlin","year":"2019"},{"key":"2023092519501002300_bib7","doi-asserted-by":"publisher","first-page":"528","DOI":"10.18653\/v1\/K19-1049","article-title":"Learning dense representations for entity retrieval","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Gillick","year":"2019"},{"key":"2023092519501002300_bib8","doi-asserted-by":"publisher","first-page":"2333","DOI":"10.1145\/2505515.2505665","article-title":"Learning deep structured semantic models for web search using clickthrough data","volume-title":"Proceedings of the 22nd ACM International Conference on Information & Knowledge Management","author":"Huang","year":"2013"},{"key":"2023092519501002300_bib9","doi-asserted-by":"publisher","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Karpukhin","year":"2020"},{"key":"2023092519501002300_bib11","article-title":"Adam: A method for stochastic optimization","volume-title":"ICLR (Poster)","author":"Kingma","year":"2015"},{"key":"2023092519501002300_bib13","doi-asserted-by":"publisher","first-page":"2147","DOI":"10.18653\/v1\/2022.findings-acl.169","article-title":"Tabi: Type-aware bi-encoders for open-domain entity retrieval","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Leszczynski","year":"2022"},{"key":"2023092519501002300_bib14","article-title":"No parameters left behind: Sensitivity guided adaptive learning rate for training large transformer models","volume-title":"International Conference on Learning Representations","author":"Liang","year":"2022"},{"key":"2023092519501002300_bib15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.510","article-title":"Super tickets in pre-trained language models: From model compression to improving generalization","author":"Liang","year":"2021","journal-title":"arXiv preprint arXiv:2105.12002"},{"key":"2023092519501002300_bib16","doi-asserted-by":"publisher","first-page":"2531","DOI":"10.1145\/3404835.3462789","article-title":"Openmatch: An open source library for neu-ir research","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Liu","year":"2021"},{"key":"2023092519501002300_bib17","doi-asserted-by":"publisher","first-page":"3449","DOI":"10.18653\/v1\/P19-1335","article-title":"Zero-shot entity linking by reading entity descriptions","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Logeswaran","year":"2019"},{"key":"2023092519501002300_bib18","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.18653\/v1\/2021.acl-long.89","article-title":"Multi-task retrieval for knowledge-intensive tasks","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Maillard","year":"2021"},{"key":"2023092519501002300_bib19","article-title":"Are sixteen heads really better than one?","volume":"32","author":"Michel","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023092519501002300_bib20","doi-asserted-by":"publisher","first-page":"11264","DOI":"10.1109\/CVPR.2019.01152","article-title":"Importance estimation for neural network pruning","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Molchanov","year":"2019"},{"key":"2023092519501002300_bib21","article-title":"Pruning convolutional neural networks for resource efficient inference","author":"Molchanov","year":"2016","journal-title":"arXiv preprint arXiv:1611.06440"},{"key":"2023092519501002300_bib22","first-page":"660","article-title":"Ms marco: A human generated machine reading comprehension dataset","volume":"2640","author":"Nguyen","year":"2016","journal-title":"Choice"},{"key":"2023092519501002300_bib23","article-title":"Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models","author":"Ni","year":"2021","journal-title":"arXiv preprint arXiv:2108.08877"},{"key":"2023092519501002300_bib24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.200","article-title":"Kilt: A benchmark for knowledge intensive language tasks","volume-title":"NAACL-HLT","author":"Petroni","year":"2021"},{"key":"2023092519501002300_bib25","article-title":"Focus on the common good: Group distributional robustness follows","author":"Piratla","year":"2021","journal-title":"arXiv preprint arXiv:2110.02619"},{"key":"2023092519501002300_bib26","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019","journal-title":"Journal of Machine Learning Research"},{"key":"2023092519501002300_bib27","article-title":"Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models","author":"Thakur","year":"2021","journal-title":"arXiv preprint arXiv:2104.08663"},{"key":"2023092519501002300_bib28","article-title":"Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models","author":"Wang","year":"2020","journal-title":"arXiv preprint arXiv:2010.05874"},{"key":"2023092519501002300_bib12","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-main.519","article-title":"Zero-shot entity linking with dense entity retrieval","volume-title":"EMNLP","author":"Wu","year":"2020"},{"key":"2023092519501002300_bib29","article-title":"Approximate nearest neighbor negative contrastive learning for dense text retrieval","volume-title":"International Conference on Learning Representations","author":"Xiong","year":"2021"},{"key":"2023092519501002300_bib30","first-page":"5824","article-title":"Gradient surgery for multi-task learning","volume":"33","author":"Tianhe","year":"2020","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00597\/2159628\/tacl_a_00597.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00597\/2159628\/tacl_a_00597.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,25]],"date-time":"2023-09-25T19:50:25Z","timestamp":1695671425000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00597\/117689\/Improving-Multitask-Retrieval-by-Promoting-Task"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":30,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00597","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}