{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T09:29:14Z","timestamp":1763458154777,"version":"3.37.3"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61572434"],"award-info":[{"award-number":["61572434"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Bidirectional encoder representations from transformers (BERT) have achieved great success in many natural language processing tasks. However, BERT generally takes the embedding of the first token to represent sentence meaning in the tasks such as sentiment analysis and textual similarity, which does not properly treat different sentence parts. Different sentence parts have different levels of importance for different downstream tasks. For example, main parts (subject, predicate, and object) play crucial roles in textual similarity calculation, while secondary parts (adverbial and complement) are more important than the main parts in sentiment analysis. To this end, we propose a sentence part-enhanced BERT (SpeBERT) model that uses sentence parts with respect to downstream tasks to enhance sentence representations. Specifically, we encode sentence parts based on dependency parsing and downstream tasks, and extract embeddings through a pooling operation. Furthermore, we design several fusion strategies to incorporate different embeddings. We evaluate the proposed SpeBERT model on two downstream tasks, sentiment classification, and semantic textual similarity, with six benchmark datasets. The experimental results show that our model achieves better performance than competitor models.<\/jats:p>","DOI":"10.1007\/s40747-022-00819-1","type":"journal-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T10:04:04Z","timestamp":1657879444000},"page":"463-474","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Sentence part-enhanced BERT with respect to downstream tasks"],"prefix":"10.1007","volume":"9","author":[{"given":"Chaoming","family":"Liu","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9656-9781","authenticated-orcid":false,"given":"Wenhao","family":"Zhu","sequence":"additional","affiliation":[]},{"given":"Xiaoyu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Qiuhong","family":"Zhai","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,7,15]]},"reference":[{"key":"819_CR1","unstructured":"Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. https:\/\/d4mucfpksywv.cloudfront.net\/better-language-models\/language-models.pdf"},{"key":"819_CR2","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova KN (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol 1 (Long and Short Papers), pp 4171\u20134186"},{"key":"819_CR3","unstructured":"Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: Generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol 32"},{"key":"819_CR4","unstructured":"Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692"},{"key":"819_CR5","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, vol 30, pp 5998\u20136008"},{"key":"819_CR6","unstructured":"Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp 1631\u20131642"},{"key":"819_CR7","doi-asserted-by":"crossref","unstructured":"Cer D, Diab M, Agirre E, Lopez-Gazpio I, Specia L (2017) Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp 1\u201314","DOI":"10.18653\/v1\/S17-2001"},{"key":"819_CR8","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2019) Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp 3980\u20133990","DOI":"10.18653\/v1\/D19-1410"},{"key":"819_CR9","unstructured":"Kusner M, Sun Y, Kolkin N, Weinberger K (2015) From word embeddings to document distances. In: Proceedings of The 32nd International Conference on Machine Learning, pp 957\u2013966"},{"key":"819_CR10","unstructured":"Kiros R, Zhu Y, Salakhutdinov R, Zemel RS, Torralba A, Urtasun R, Fidler S (2015) Skip-thought vectors. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol 28, pp 3294\u20133302"},{"key":"819_CR11","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol 26, pp 3111\u20133119"},{"key":"819_CR12","doi-asserted-by":"crossref","unstructured":"Hill F, Cho K, Korhonen A (2016) Learning distributed representations of sentences from unlabelled data. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 1367\u20131377","DOI":"10.18653\/v1\/N16-1162"},{"key":"819_CR13","unstructured":"Lin Z, Feng M, dos Santos CN, Yu M, Xiang B, Zhou B, Bengio Y (2017) A structured self-attentive sentence embedding. In: ICLR 2017 : International Conference on Learning Representations 2017"},{"key":"819_CR14","unstructured":"Arora S, Liang Y, Ma T (2017) A simple but tough-to-beat baseline for sentence embeddings. In: ICLR 2017 : International Conference on Learning Representations 2017"},{"key":"819_CR15","doi-asserted-by":"crossref","unstructured":"Conneau A, Kiela D, Schwenk H, Barrault L, Bordes A (2017) Supervised learning of universal sentence representations from natural language inference data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 670\u2013680","DOI":"10.18653\/v1\/D17-1070"},{"key":"819_CR16","doi-asserted-by":"crossref","unstructured":"Bowman SR, Angeli G, Potts C, Manning CD (2015) A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp 632\u2013642","DOI":"10.18653\/v1\/D15-1075"},{"key":"819_CR17","doi-asserted-by":"crossref","unstructured":"Yang Y, Yuan S, Cer D, Kong S-y, Constant N, Pilar P, Ge H, Sung Y-H, Strope B, Kurzweil R (2018) Learning semantic textual similarity from conversations. In: Proceedings of The Third Workshop on Representation Learning for NLP, Association for Computational Linguistics, Melbourne, Australia, pp 164\u2013174","DOI":"10.18653\/v1\/W18-3022"},{"key":"819_CR18","doi-asserted-by":"crossref","unstructured":"Cer D, Yang Y, Kong S-y, Hua N, Limtiaco N, John RS, Constant N, Guajardo-Cespedes M, Yuan S, Tar C, Sung Y-H, Strope B, Kurzweil R (2018) Universal sentence encoder. arXiv preprint arXiv:1803.11175","DOI":"10.18653\/v1\/D18-2029"},{"key":"819_CR19","doi-asserted-by":"crossref","unstructured":"Ethayarajh K (2019) How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp 55\u201365","DOI":"10.18653\/v1\/D19-1006"},{"key":"819_CR20","doi-asserted-by":"crossref","unstructured":"Li B, Zhou H, He J, Wang M, Yang Y, Li L (2020) On the sentence embeddings from pre-trained language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online","DOI":"10.18653\/v1\/2020.emnlp-main.733"},{"key":"819_CR21","doi-asserted-by":"crossref","unstructured":"Zhu W, Jin X, Liu S, Lu Z, Zhang W, Yan K, Wei B (2020) Enhanced double-carrier word embedding via phonetics and writing. ACM Transactions on Asian and Low-Resource Language Information Processing 19(2)","DOI":"10.1145\/3344920"},{"issue":"6","key":"819_CR22","doi-asserted-by":"publisher","first-page":"102709","DOI":"10.1016\/j.ipm.2021.102709","volume":"58","author":"W Zhu","year":"2021","unstructured":"Zhu W, Liu S, Liu C (2021) Learning multimodal word representation with graph convolutional networks. Information Processing & Management 58(6):102709","journal-title":"Information Processing & Management"},{"key":"819_CR23","unstructured":"Subramanian S, Trischler A, Bengio Y, Pal CJ (2018) Learning general purpose distributed sentence representations via large scale multi-task learning. In: International Conference on Learning Representations"},{"key":"819_CR24","doi-asserted-by":"crossref","unstructured":"Nie A, Bennett E, Goodman ND (2019) Dissent: Learning sentence representations from explicit discourse relations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 4497\u20134510","DOI":"10.18653\/v1\/P19-1442"},{"key":"819_CR25","doi-asserted-by":"crossref","unstructured":"Chen D, Manning C (2014) A fast and accurate dependency parser using neural networks. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp 740\u2013750","DOI":"10.3115\/v1\/D14-1082"},{"key":"819_CR26","doi-asserted-by":"crossref","unstructured":"Pang B, Lee L (2005) Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pp 115\u2013124","DOI":"10.3115\/1219840.1219855"},{"key":"819_CR27","doi-asserted-by":"crossref","unstructured":"Hu M, Liu B (2004) Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 168\u2013177","DOI":"10.1145\/1014052.1014073"},{"key":"819_CR28","unstructured":"Marelli M, Menini S, Baroni M, Bentivogli L, bernardi R, Zamparelli R (2014) A sick cure for the evaluation of compositional distributional semantic models. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914), pp 216\u2013223"},{"key":"819_CR29","unstructured":"Zeng J (2020) A large-scale chinese nature language inference and semantic similarity calculation dataset. https:\/\/6a75-junzeng-uxxxm-1300734931.tcb.qcloud.la\/CNSD.pdf"},{"key":"819_CR30","unstructured":"Kingma DP, Ba JL (2015) Adam: A method for stochastic optimization. In: International Conference on Learning Representations"},{"key":"819_CR31","doi-asserted-by":"crossref","unstructured":"Tai KS, Socher R, Manning CD (2015) Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, vol 1 (Long Papers), pp 1556\u20131566","DOI":"10.3115\/v1\/P15-1150"},{"key":"819_CR32","doi-asserted-by":"crossref","unstructured":"Yan Y, Li R, Wang S, Zhang F, Wu W, Xu W (2021) ConSERT: A contrastive framework for self-supervised sentence representation transfer. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, pp 5065\u20135075","DOI":"10.18653\/v1\/2021.acl-long.393"},{"key":"819_CR33","doi-asserted-by":"crossref","unstructured":"Gao T, Yao X, Chen D (2021) SimCSE: Simple contrastive learning of sentence embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, pp 6894\u20136910","DOI":"10.18653\/v1\/2021.emnlp-main.552"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00819-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-022-00819-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00819-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,22]],"date-time":"2023-02-22T18:55:49Z","timestamp":1677092149000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-022-00819-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,15]]},"references-count":33,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["819"],"URL":"https:\/\/doi.org\/10.1007\/s40747-022-00819-1","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2022,7,15]]},"assertion":[{"value":"14 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"As the current research is still in progress, we decided not to share the code for the time being.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}}]}}