{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,12,16]],"date-time":"2023-12-16T00:48:11Z","timestamp":1702687691659},"reference-count":46,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T00:00:00Z","timestamp":1702598400000},"content-version":"vor","delay-in-days":348,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,12,14]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. One straightforward approach is to finetune a pretrained language model (PLM) with a representation learning method such as contrastive learning. While this approach achieves impressive performance on larger PLMs, the performance rapidly degrades as the number of parameters decreases. In this paper, we propose a framework called Self-supervised Cross-View Training (SCT) to narrow the performance gap between large and small PLMs. To evaluate the effectiveness of SCT, we compare it to 5 baseline and state-of-the-art competitors on seven Semantic Textual Similarity (STS) benchmarks using 5 PLMs with the number of parameters ranging from 4M to 340M. The experimental results show that STC outperforms the competitors for PLMs with less than 100M parameters in 18 of 21 cases.1<\/jats:p>","DOI":"10.1162\/tacl_a_00620","type":"journal-article","created":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T18:58:58Z","timestamp":1702666738000},"page":"1572-1587","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["An Efficient Self-Supervised Cross-View Training For Sentence Embedding"],"prefix":"10.1162","volume":"11","author":[{"given":"Peerat","family":"Limkonchotiwat","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, VISTEC, Thailand. peerat.l_s19@vistec.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wuttikorn","family":"Ponwitayarat","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, VISTEC, Thailand. wuttikorn.p_s22@vistec.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lalita","family":"Lowphansirikul","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, VISTEC, Thailand. lalita.l_s22@vistec.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Can","family":"Udomcharoenchaikit","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, VISTEC, Thailand. canu_pro@vistec.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ekapol","family":"Chuangsuwanich","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Chulalongkorn University, Thailand. ekapolc@cp.eng.chula.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sarana","family":"Nutanong","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, VISTEC, Thailand. snutanon@vistec.ac.th"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,12,14]]},"reference":[{"key":"2023121518584497000_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S15-2045","article-title":"SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability","volume-title":"SemEval 2015","author":"Agirre","year":"2015"},{"key":"2023121518584497000_bib2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/S14-2010","article-title":"SemEval-2014 task 10: Multilingual semantic textual similarity","volume-title":"SemEval 2014","author":"Agirre","year":"2014"},{"key":"2023121518584497000_bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S16-1081","article-title":"SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation","volume-title":"SemEval-2016","author":"Agirre","year":"2016"},{"key":"2023121518584497000_bib4","article-title":"SemEval-2012 task 6: A pilot on semantic textual similarity","volume-title":"*SEM 2012: The First Joint Conference on Lexical and Computational Semantics \u2013 Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)","author":"Agirre","year":"2012"},{"key":"2023121518584497000_bib5","article-title":"*SEM 2013 shared task: Semantic textual similarity","volume-title":"Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity","author":"Agirre","year":"2013"},{"key":"2023121518584497000_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1075","article-title":"A large annotated corpus for learning natural language inference","volume-title":"EMNLP 2015","author":"Bowman","year":"2015"},{"key":"2023121518584497000_bib7","first-page":"822","article-title":"Why do larger models generalize better? A theoretical perspective via the XOR problem","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Brutzkus","year":"2019"},{"key":"2023121518584497000_bib8","doi-asserted-by":"publisher","first-page":"3138","DOI":"10.18653\/v1\/2022.findings-acl.248","article-title":"Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Cao","year":"2022"},{"key":"2023121518584497000_bib9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-2001","article-title":"SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation","volume-title":"Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)","author":"Cer","year":"2017"},{"key":"2023121518584497000_bib10","article-title":"Improved baselines with momentum contrastive learning","author":"Chen","year":"2020"},{"key":"2023121518584497000_bib11","doi-asserted-by":"publisher","first-page":"15750","DOI":"10.1109\/CVPR46437.2021.01549","article-title":"Exploring simple Siamese representation learning","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Chen","year":"2021"},{"key":"2023121518584497000_bib12","doi-asserted-by":"publisher","first-page":"4207","DOI":"10.18653\/v1\/2022.naacl-main.311","article-title":"DiffCSE: Difference- based contrastive learning for sentence embeddings","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Chuang","year":"2022"},{"key":"2023121518584497000_bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.207","article-title":"Specter: Document-level representation learning using citation-informed transformers","volume-title":"ACL","author":"Cohan","year":"2020"},{"key":"2023121518584497000_bib14","article-title":"Senteval: An evaluation toolkit for universal sentence representations","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7\u201312, 2018","author":"Conneau","year":"2018"},{"key":"2023121518584497000_bib15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"NAACL-HLT 2019","author":"Devlin","year":"2019"},{"key":"2023121518584497000_bib16","doi-asserted-by":"publisher","DOI":"10.36227\/techrxiv.12308378.v1","article-title":"Cert: Contrastive self-supervised learning for language understanding","author":"Fang","year":"2020"},{"key":"2023121518584497000_bib17","article-title":"SEED: Self-supervised distillation for visual representation","volume-title":"ICLR 2021","author":"Fang","year":"2021"},{"key":"2023121518584497000_bib18","doi-asserted-by":"publisher","first-page":"6894","DOI":"10.18653\/v1\/2021.emnlp-main.552","article-title":"SimCSE: Simple contrastive learning of sentence embeddings","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Gao","year":"2021"},{"key":"2023121518584497000_bib19","first-page":"21271","article-title":"Bootstrap your own latent\u2014A new approach to self-supervised learning","volume-title":"Advances in Neural Information Processing Systems","author":"Grill","year":"2020"},{"key":"2023121518584497000_bib20","doi-asserted-by":"publisher","first-page":"9726","DOI":"10.1109\/CVPR42600.2020.00975","article-title":"Momentum contrast for unsupervised visual representation learning","volume-title":"2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"He","year":"2020"},{"key":"2023121518584497000_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.197","article-title":"Self-guided contrastive learning for BERT sentence representations","volume-title":"ACL\/IJCNLP 2021","author":"Kim","year":"2021"},{"key":"2023121518584497000_bib22","doi-asserted-by":"publisher","first-page":"394","DOI":"10.18653\/v1\/2022.acl-short.44","article-title":"SCD: Self-contrastive decorrelation of sentence embeddings","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Klein","year":"2022"},{"key":"2023121518584497000_bib23","doi-asserted-by":"publisher","first-page":"1279","DOI":"10.18653\/v1\/N16-1153","article-title":"Semi-supervised question retrieval with gated convolutions","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Lei","year":"2016"},{"key":"2023121518584497000_bib24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.733","article-title":"On the sentence embeddings from pre- trained language models","volume-title":"EMNLP 2020","author":"Li","year":"2020"},{"key":"2023121518584497000_bib25","doi-asserted-by":"publisher","first-page":"6467","DOI":"10.18653\/v1\/2022.findings-emnlp.483","article-title":"ConGen: Unsupervised control and generalization distillation for sentence representation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Limkonchotiwat","year":"2022"},{"key":"2023121518584497000_bib26","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.109","article-title":"Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders","volume-title":"EMNLP, 2021","author":"Liu","year":"2021"},{"key":"2023121518584497000_bib27","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1907.11692","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu","year":"2019"},{"key":"2023121518584497000_bib28","article-title":"Decoupled weight decay regularization","volume-title":"7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\u20139, 2019","author":"Loshchilov","year":"2019"},{"key":"2023121518584497000_bib29","first-page":"216","article-title":"A SICK cure for the evaluation of compositional distributional semantic models","volume-title":"Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914)","author":"Marelli","year":"2014"},{"key":"2023121518584497000_bib30","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"key":"2023121518584497000_bib31","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1910.01108","article-title":"Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter","author":"Sanh","year":"2019"},{"key":"2023121518584497000_bib32","doi-asserted-by":"publisher","first-page":"14411","DOI":"10.1109\/CVPR52688.2022.01403","article-title":"Exploring the equivalence of Siamese self-supervised learning via a unified gradient framework","volume-title":"IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18\u201324, 2022","author":"Tao","year":"2022"},{"key":"2023121518584497000_bib33","doi-asserted-by":"publisher","first-page":"9589","DOI":"10.1109\/ICCV48922.2021.00947","article-title":"Isd: Self-supervised learning by iterative similarity distillation","volume-title":"2021 IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Tejankar","year":"2021"},{"key":"2023121518584497000_bib34","article-title":"Well-read students learn better: On the importance of pre-training compact models","author":"Turc","year":"2019","journal-title":"arXiv preprint arXiv:1908.08962v2"},{"key":"2023121518584497000_bib35","doi-asserted-by":"publisher","first-page":"671","DOI":"10.18653\/v1\/2021.findings-emnlp.59","article-title":"TSDAE: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Wang","year":"2021"},{"key":"2023121518584497000_bib36","article-title":"Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers","volume-title":"NeurIPS 2020","author":"Wang","year":"2020"},{"key":"2023121518584497000_bib37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1101","article-title":"A broad-coverage challenge corpus for sentence understanding through inference","volume-title":"NAACL 2018","author":"Williams","year":"2018"},{"key":"2023121518584497000_bib38","article-title":"Disco: Effective knowledge distillation for contrastive learning of sentence embeddings","author":"Xing","year":"2021"},{"key":"2023121518584497000_bib39","doi-asserted-by":"publisher","first-page":"5065","DOI":"10.18653\/v1\/2021.acl-long.393","article-title":"ConSERT: A contrastive framework for self-supervised sentence representation transfer","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Yan","year":"2021"},{"key":"2023121518584497000_bib40","first-page":"12310","article-title":"Barlow twins: Self-supervised learning via redundancy reduction","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Zbontar","year":"2021"},{"key":"2023121518584497000_bib41","article-title":"How does simsiam avoid collapse without negative samples? A unified understanding with self-supervised contrastive learning","volume-title":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\u201329, 2022","author":"Zhang","year":"2022"},{"key":"2023121518584497000_bib42","doi-asserted-by":"publisher","first-page":"864","DOI":"10.18653\/v1\/2022.findings-acl.70","article-title":"Virtual augmentation supported contrastive learning of sentence representations","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Zhang","year":"2022"},{"key":"2023121518584497000_bib43","doi-asserted-by":"publisher","first-page":"5168","DOI":"10.18653\/v1\/2021.acl-long.402","article-title":"Bootstrapped unsupervised sentence representation learning","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Zhang","year":"2021"},{"key":"2023121518584497000_bib44","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.124","article-title":"An unsupervised sentence embedding method by mutual information maximization","volume-title":"EMNLP 2020","author":"Zhang","year":"2020"},{"key":"2023121518584497000_bib45","first-page":"2543","article-title":"Ressl: Relational self-supervised learning with weak augmentation","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6\u201314, 2021, virtual","author":"Zheng","year":"2021"},{"key":"2023121518584497000_bib46","doi-asserted-by":"publisher","first-page":"6120","DOI":"10.18653\/v1\/2022.acl-long.423","article-title":"Debiased contrastive learning of unsupervised sentence representations","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Zhou","year":"2022"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00620\/2196817\/tacl_a_00620.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00620\/2196817\/tacl_a_00620.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T18:59:12Z","timestamp":1702666752000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00620\/118714\/An-Efficient-Self-Supervised-Cross-View-Training"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":46,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00620","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}