{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T04:28:00Z","timestamp":1785472080975,"version":"3.56.0"},"reference-count":55,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT<jats:sub>large<\/jats:sub>, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. <jats:sup>1<\/jats:sup> <\/jats:p>","DOI":"10.1162\/tacl_a_00300","type":"journal-article","created":{"date-parts":[[2020,3,12]],"date-time":"2020-03-12T14:05:38Z","timestamp":1584021938000},"page":"64-77","source":"Crossref","is-referenced-by-count":936,"title":["SpanBERT: Improving Pre-training by Representing and Predicting Spans"],"prefix":"10.1162","volume":"8","author":[{"given":"Mandar","family":"Joshi","sequence":"first","affiliation":[{"name":"Allen School of Computer Science & Engineering, University of Washington, Seattle, WA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danqi","family":"Chen","sequence":"additional","affiliation":[{"name":"Computer Science Department, Princeton University, Princeton, NJ."},{"name":"Facebook AI Research, Seattle."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinhan","family":"Liu","sequence":"additional","affiliation":[{"name":"Facebook AI Research, Seattle."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel S.","family":"Weld","sequence":"additional","affiliation":[{"name":"Allen School of Computer Science & Engineering, University of Washington, Seattle, WA."},{"name":"Allen Institute of Artificial Intelligence, Seattle."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luke","family":"Zettlemoyer","sequence":"additional","affiliation":[{"name":"Allen School of Computer Science & Engineering, University of Washington, Seattle, WA."},{"name":"Facebook AI Research, Seattle."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omer","family":"Levy","sequence":"additional","affiliation":[{"name":"Facebook AI Research, Seattle."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","author":"Ba Jimmy Lei","year":"2016","journal-title":"arXiv preprint arXiv:1607.06450"},{"key":"bib2","first-page":"6","volume-title":"Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment","author":"Bar-Haim Roy","year":"2006"},{"key":"bib3","first-page":"1","volume-title":"International Workshop on Semantic Evaluation (SemEval)","author":"Cer Daniel","year":"2017"},{"key":"bib4","author":"Chan William","year":"2019","journal-title":"arXiv preprint arXiv:1906.01604"},{"key":"bib5","first-page":"177","volume-title":"Machine Learning Challenges Workshop","author":"Dagan Ido","year":"2005"},{"key":"bib6","first-page":"3079","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Dai Andrew M.","year":"2015"},{"key":"bib7","volume-title":"Association for Computational Linguistics (ACL)","author":"Dai Zihang","year":"2019"},{"key":"bib8","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Devlin Jacob","year":"2019"},{"key":"bib9","volume-title":"Proceedings of the International Workshop on Paraphrasing","author":"Dolan William B.","year":"2005"},{"key":"bib10","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Li Dong","year":"2019"},{"key":"bib11","author":"Dunn Matthew","year":"2017","journal-title":"arXiv preprint arXiv:1704.05179"},{"key":"bib12","volume-title":"Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP","author":"Fisch Adam","year":"2019"},{"key":"bib13","first-page":"1","volume-title":"Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing","author":"Giampiccolo Danilo","year":"2007"},{"key":"bib14","first-page":"364","volume-title":"Association for Computational Linguistics (ACL)","author":"He Luheng","year":"2018"},{"key":"bib15","author":"Hendrycks Dan","year":"2016","journal-title":"arXiv preprint arXiv:1606.08415"},{"key":"bib16","unstructured":"Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear."},{"key":"bib17","author":"Howard Jeremy","year":"2018","journal-title":"arXiv preprint arXiv:1801.06146"},{"key":"bib18","first-page":"3597","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Joshi Mandar","year":"2019"},{"key":"bib19","first-page":"1601","volume-title":"Association for Computational Linguistics (ACL)","author":"Joshi Mandar","year":"2017"},{"key":"bib20","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Joshi Mandar","year":"2019"},{"key":"bib21","volume-title":"International Conference on Learning Representations (ICLR)","author":"Kingma Diederik","year":"2015"},{"key":"bib22","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Kiros Ryan","year":"2015"},{"key":"bib23","author":"Kwiatkowski Tom","year":"2019","journal-title":"Transactions of the Association of Computational Linguistics (TACL)"},{"key":"bib24","author":"Lample Guillaume","year":"2019","journal-title":"Advances in Neural Information Processing Systems (NIPS)"},{"key":"bib25","first-page":"188","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Lee Kenton","year":"2017"},{"key":"bib26","first-page":"687","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Lee Kenton","year":"2018"},{"key":"bib27","author":"Lee Kenton","year":"2016","journal-title":"arXiv preprint arXiv:1611.01436"},{"key":"bib28","first-page":"47","volume-title":"AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning","volume":"46","author":"Levesque Hector J.","year":"2011"},{"key":"bib29","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Liu Xiaodong","year":"2019"},{"key":"bib30","author":"Liu Yinhan","year":"2019","journal-title":"arxiv preprint arXiv:1907.11692"},{"key":"bib31","author":"Logeswaran Lajanugen","year":"2018","journal-title":"arxiv preprint arXiv:1803.02893"},{"key":"bib32","volume-title":"International Conference on Learning Representations (ICLR)","author":"Loshchilov Ilya","year":"2019"},{"key":"bib33","first-page":"51","volume-title":"Computational Natural Language Learning (CoNLL)","author":"Melamud Oren","year":"2016"},{"key":"bib34","first-page":"48","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Ott Myle","year":"2019"},{"key":"bib35","first-page":"2227","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Peters Matthew","year":"2018"},{"key":"bib36","first-page":"1","volume-title":"Joint Conference on EMNLP and CoNLL-Shared Task","author":"Pradhan Sameer","year":"2012"},{"key":"bib37","first-page":"157","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers","author":"Press Ofir","year":"2017"},{"key":"bib38","unstructured":"Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning, OpenAI."},{"key":"bib39","first-page":"784","volume-title":"Association for Computational Linguistics (ACL)","author":"Rajpurkar Pranav","year":"2018"},{"key":"bib40","first-page":"2383","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Rajpurkar Pranav","year":"2016"},{"key":"bib41","first-page":"2895","volume-title":"Association for Computational Linguistics (ACL)","author":"Soares Livio Baldini","year":"2019"},{"key":"bib42","first-page":"1631","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Socher Richard","year":"2013"},{"key":"bib43","first-page":"5926","volume-title":"International Conference on Machine Learning (ICML)","author":"Song Kaitao","year":"2019"},{"key":"bib44","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Stern Mitchell","year":"2018"},{"key":"bib45","author":"Stephanie Sun Yu","year":"2019","journal-title":"arXiv preprint arXiv:1904.09223"},{"key":"bib46","doi-asserted-by":"crossref","first-page":"191","DOI":"10.18653\/v1\/W17-2623","volume-title":"2nd Workshop on Representation Learning for NLP","author":"Trischler Adam","year":"2017"},{"key":"bib47","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Vaswani Ashish","year":"2017"},{"key":"bib48","volume-title":"International Conference on Learning Representations (ICLR)","author":"Wang Alex","year":"2019"},{"key":"bib49","author":"Warstadt Alex","year":"2018","journal-title":"arXiv preprint arXiv:1805.12471"},{"key":"bib50","first-page":"1112","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Williams Adina","year":"2018"},{"key":"bib51","author":"Wolf Thomas","year":"2019","journal-title":"arXiv preprint arXiv:1910.03771"},{"key":"bib52","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Yang Zhilin","year":"2019"},{"key":"bib53","first-page":"2369","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Yang Zhilin","year":"2018"},{"key":"bib54","first-page":"35","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Zhang Yuhao","year":"2017"},{"key":"bib55","first-page":"1441","volume-title":"Association for Computational Linguistics (ACL)","author":"Zhang Zhengyan","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00300","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:32Z","timestamp":1615585172000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43539"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":55,"alternative-id":["10.1162\/tacl_a_00300"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00300","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}