{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T00:28:51Z","timestamp":1777854531558,"version":"3.51.4"},"reference-count":74,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2021,2,15]],"date-time":"2021-02-15T00:00:00Z","timestamp":1613347200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Science"],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:p>Text representation is a fundamental cornerstone that impacts the effectiveness of several text summarization methods. Transfer learning using pre-trained word embedding models has shown promising results. However, most of these representations do not consider the order and the semantic relationships between words in a sentence, and thus they do not carry the meaning of a full sentence. To overcome this issue, the current study proposes an unsupervised method for extractive multi-document summarization based on transfer learning from BERT sentence embedding model. Moreover, to improve sentence representation learning, we fine-tune BERT model on supervised intermediate tasks from GLUE benchmark datasets using single-task and multi-task fine-tuning methods. Experiments are performed on the standard DUC\u20192002\u20132004 datasets. The obtained results show that our method has significantly outperformed several baseline methods and achieves a comparable and sometimes better performance than the recent state-of-the-art deep learning\u2013based methods. Furthermore, the results show that fine-tuning BERT using multi-task learning has considerably improved the performance.<\/jats:p>","DOI":"10.1177\/0165551521990616","type":"journal-article","created":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T17:06:38Z","timestamp":1613840798000},"page":"164-182","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":44,"title":["Unsupervised extractive multi-document summarization method based on transfer learning from BERT multi-task fine-tuning"],"prefix":"10.1177","volume":"49","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8789-5713","authenticated-orcid":false,"given":"Salima","family":"Lamsiyah","sequence":"first","affiliation":[{"name":"Laboratory of Informatics, Signals, Automatic, and Cognitivism, FSDM, Sidi Mohamed Ben Abdellah University, Morocco; Laboratory of Engineering Sciences, National School of Applied Sciences, Ibn Tofail University, Morocco"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4281-2472","authenticated-orcid":false,"given":"Abdelkader El","family":"Mahdaouy","sequence":"additional","affiliation":[{"name":"School of Computer Science (UM6P-CS), Mohammed VI Polytechnic University (UM6P), Morocco"}]},{"given":"Sa\u00efd El Alaoui","family":"Ouatik","sequence":"additional","affiliation":[{"name":"Laboratory of Informatics, Signals, Automatic, and Cognitivism, FSDM, Sidi Mohamed Ben Abdellah University, Morocco; Laboratory of Engineering Sciences, National School of Applied Sciences, Ibn Tofail University, Morocco"}]},{"given":"Bernard","family":"Espinasse","sequence":"additional","affiliation":[{"name":"LIS UMR CNRS 7020, Aix-Marseille Universit\u00e9\/Universit\u00e9 de Toulon, France"}]}],"member":"179","published-online":{"date-parts":[[2021,2,15]]},"reference":[{"key":"bibr1-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1147\/rd.22.0159"},{"key":"bibr2-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2003.10.006"},{"key":"bibr3-0165551521990616","first-page":"1188","volume-title":"Proceedings of the 31th international conference on machine learning, ICML","author":"Le QV"},{"key":"bibr4-0165551521990616","first-page":"3111","volume-title":"Advances in neural information processing systems","author":"Mikolov T"},{"key":"bibr5-0165551521990616","first-page":"1532","volume-title":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","author":"Pennington J"},{"key":"bibr6-0165551521990616","first-page":"1984","volume-title":"Proceedings of the 2015 conference on empirical methods in natural language processing (EMNLP)","author":"Kobayashi H"},{"key":"bibr7-0165551521990616","volume-title":"2017 international conference on machine learning and data science (MLDS)","author":"Jain A"},{"key":"bibr8-0165551521990616","volume-title":"Proceedings of the MultiLing 2017 workshop on summarization and summary evaluation across source types and genres","author":"Rossiello G"},{"key":"bibr9-0165551521990616","first-page":"85","volume-title":"Proceedings of the Workshop on New Frontiers in Summarization, Nfis@emnlp","author":"Ghalandari DG"},{"key":"bibr10-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2019.112958"},{"key":"bibr11-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1177\/0165551518786676"},{"key":"bibr12-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1177\/0165551518792210"},{"key":"bibr13-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007379606734"},{"key":"bibr14-0165551521990616","first-page":"912","volume-title":"Proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies","author":"Liu X"},{"key":"bibr15-0165551521990616","volume-title":"Proceedings of the 4th international conference on learning representations","author":"Luong M"},{"key":"bibr16-0165551521990616","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1064"},{"key":"bibr17-0165551521990616","volume-title":"Proceedings of the 6th International Conference on Learning Representations, ICLR","author":"Subramanian S"},{"key":"bibr18-0165551521990616","first-page":"4822","volume":"33","author":"Ruder S","year":"2019","journal-title":"Proc AAAI Conf Artif Intel"},{"key":"bibr19-0165551521990616","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL-HLT","author":"Devlin J"},{"key":"bibr20-0165551521990616","unstructured":"Kiros R, Zhu Y, Salakhutdinov RR, et al. Skip-thought vectors. In: Advances in neural information processing systems, pp. 3294\u20133302, https:\/\/arxiv.org\/abs\/1506.06726"},{"key":"bibr21-0165551521990616","first-page":"2227","volume-title":"Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL-HLT","author":"Peters ME"},{"key":"bibr22-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2019.03.045"},{"key":"bibr23-0165551521990616","first-page":"353","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wang A"},{"key":"bibr24-0165551521990616","first-page":"74","volume-title":"Text Summarization Branches Out. Association for Computational Linguistics","author":"Lin CY"},{"key":"bibr25-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1177\/0165551511408848"},{"key":"bibr26-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2013.04.023"},{"key":"bibr27-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2016.08.030"},{"key":"bibr28-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1177\/0165551517693537"},{"key":"bibr29-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1177\/0165551519827896"},{"key":"bibr30-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-017-1042-4"},{"key":"bibr31-0165551521990616","unstructured":"Aries A, Zegour DE, Hidouci W. Automatic text summarization: What has been done and what has to be done, 2019, https:\/\/arxiv.org\/abs\/1904.00688v1"},{"key":"bibr32-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2015.05.034"},{"key":"bibr33-0165551521990616","unstructured":"Cao Z, Wei F, Dong L, et al. Ranking with recursive neural networks and its application to multi-document summarization. In: Twenty-ninth AAAI conference on artificial intelligence, https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI15\/paper\/view\/9414"},{"key":"bibr34-0165551521990616","first-page":"452","volume-title":"Proceedings of the 21st conference on computational natural language learning (Conll)","author":"Yasunaga M"},{"key":"bibr35-0165551521990616","first-page":"4131","volume-title":"Proceedings of the 2018 conference on empirical methods in natural language processing","author":"Lebanoff L"},{"key":"bibr36-0165551521990616","unstructured":"Denil M, Demiraj A, de Freitas N. Extraction of salient sentences from labelled documents, 2014, https:\/\/arxiv.org\/abs\/1412.6815"},{"key":"bibr37-0165551521990616","first-page":"1383","volume-title":"Proceedings of the twenty-fourth international joint conference on artificial intelligence, IJCAI","author":"Yin W"},{"key":"bibr38-0165551521990616","first-page":"829","volume-title":"Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing","author":"Cao Z"},{"key":"bibr39-0165551521990616","first-page":"31","volume-title":"Proceedings of the 2nd workshop on continuous vector space models and their compositionality (CVSC)","author":"K\u00e5geb\u00e4ck M"},{"key":"bibr40-0165551521990616","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1228"},{"key":"bibr41-0165551521990616","first-page":"705","volume-title":"2017 IEEE global conference on signal and information processing (GlobalSIP)","author":"Zhang C"},{"key":"bibr42-0165551521990616","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop","author":"Bouscarrat L"},{"key":"bibr43-0165551521990616","first-page":"3730","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9Th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Liu Y"},{"key":"bibr44-0165551521990616","first-page":"670","volume-title":"Proceedings of the 2017 conference on empirical methods in natural language processing, EMNLP","author":"Conneau A"},{"key":"bibr45-0165551521990616","first-page":"1923","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Hashimoto K"},{"key":"bibr46-0165551521990616","unstructured":"Jernite Y, Bowman SR, Sontag DA. Discourse-based objectives for fast unsupervised sentence representation learning, 2017, https:\/\/arxiv.org\/abs\/1705.00557"},{"key":"bibr47-0165551521990616","first-page":"4487","volume-title":"Proceedings of the 57th conference of the association for computational linguistics, ACL","author":"Liu X"},{"key":"bibr48-0165551521990616","unstructured":"Liu X, He P, Chen W, et al. Improving multi-task deep neural networks via knowledge distillation for natural language understanding, 2019, https:\/\/arxiv.org\/abs\/1904.09482v1"},{"key":"bibr49-0165551521990616","unstructured":"Hinton GE, Vinyals O, Dean J. Distilling the knowledge in a neural network, 2015, https:\/\/arxiv.org\/abs\/1503.02531"},{"key":"bibr50-0165551521990616","unstructured":"Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Advances in neural information processing systems, pp. 5998\u20136008, https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"bibr51-0165551521990616","first-page":"19","volume-title":"Proceedings of the IEEE international conference on computer vision","author":"Zhu Y"},{"key":"bibr52-0165551521990616","first-page":"194","volume-title":"china national conference on Chinese computational linguistics","author":"Sun C"},{"key":"bibr53-0165551521990616","first-page":"1631","volume-title":"Proceedings of the 2013 conference on empirical methods in natural language processing","author":"Socher R"},{"key":"bibr54-0165551521990616","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1157"},{"key":"bibr55-0165551521990616","first-page":"1","volume-title":"Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017)","author":"Cer D"},{"key":"bibr56-0165551521990616","first-page":"2383","volume-title":"Proceedings of the 2016 conference on empirical methods in natural language processing","author":"Rajpurkar P"},{"key":"bibr57-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00290"},{"key":"bibr58-0165551521990616","unstructured":"Bentivogli L, Dagan IK, Hoa D, et al. The fifth Pascal recognizing textual entailment challenge. In: TAC 2009 workshop, http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.232.1231&rep=rep1&type=pdf"},{"key":"bibr59-0165551521990616","first-page":"1112","volume-title":"Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies","author":"Williams A"},{"key":"bibr60-0165551521990616","unstructured":"Dolan WB, Brockett C. Automatically constructing a corpus of sentential paraphrases. In: Proceedings of the third international workshop on paraphrasing (IWP2005), https:\/\/www.aclweb.org\/anthology\/I05-5002\/"},{"key":"bibr61-0165551521990616","unstructured":"Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, ICLR, 2015, https:\/\/arxiv.org\/abs\/1412.6980"},{"key":"bibr62-0165551521990616","volume-title":"The international conference on language resources and evaluation LREC\u201914","author":"Hong K"},{"key":"bibr63-0165551521990616","volume-title":"Proceedings of the eleventh international conference on language resources and evaluation (LREC-2018)","author":"Dernoncourt F"},{"key":"bibr64-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1523"},{"key":"bibr65-0165551521990616","first-page":"362","volume-title":"Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics","author":"Haghighi A"},{"key":"bibr66-0165551521990616","unstructured":"Conroy JM, Goldstein J, Schlesinger JD, et al. Left-brain\/right-brain multi-document summarization. In Proceedings of the Document Understanding Conference DUC\u2019, 2004, http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.148.1457"},{"key":"bibr67-0165551521990616","volume-title":"Proceedings of the First Text Analysis Conference, TAC","author":"Gillick D"},{"key":"bibr68-0165551521990616","first-page":"454","volume-title":"IEEE 12th International Conference on Data Mining Workshops, ICDM","author":"Davis ST"},{"key":"bibr69-0165551521990616","first-page":"712","volume-title":"Proceedings of the 14th conference of the European chapter of the association for computational linguistics","author":"Hong K"},{"key":"bibr70-0165551521990616","first-page":"479","volume-title":"Proceedings of the Twenty-Eighth conference on uncertainty in artificial intelligence","author":"Lin H"},{"key":"bibr71-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1561\/2200000044"},{"key":"bibr72-0165551521990616","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/S14-1010"},{"key":"bibr73-0165551521990616","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324918000414"},{"key":"bibr74-0165551521990616","unstructured":"Phang J, F\u00e9vry T, Bowman SR. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks, 2018, https:\/\/arxiv.org\/abs\/1811.01088"}],"container-title":["Journal of Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0165551521990616","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0165551521990616","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0165551521990616","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T23:09:31Z","timestamp":1777504171000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0165551521990616"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,15]]},"references-count":74,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["10.1177\/0165551521990616"],"URL":"https:\/\/doi.org\/10.1177\/0165551521990616","relation":{},"ISSN":["0165-5515","1741-6485"],"issn-type":[{"value":"0165-5515","type":"print"},{"value":"1741-6485","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,15]]}}}