{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T02:02:35Z","timestamp":1783476155884,"version":"3.55.0"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T00:00:00Z","timestamp":1719273600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62376033, 62222212, 62232006, 62121002"],"award-info":[{"award-number":["62376033, 62222212, 62232006, 62121002"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>\n            Document-level relation extraction (RE) aims to simultaneously predict relations (including no-relation cases denoted as NA) between all entity pairs in a document. It is typically formulated as a relation classification task with entities pre-detected in advance and solved by a hard-label training regime, which, however, neglects the divergence of the NA class and the correlations among other classes. This article introduces\n            <jats:bold>progressive self-distillation<\/jats:bold>\n            (PSD), a new training regime that employs online, self-knowledge distillation (KD) to produce and incorporate soft labels for document-level RE.The key idea of PSD is to gradually soften hard labels using past predictions from an RE model itself, which are adjusted adaptively as training proceeds. As such, PSD has to learn only one RE model within a single training pass, requiring no extra computation or annotation to pretrain another high-capacity teacher. PSD is conceptually simple, easy to implement, and generally applicable to various RE models to further improve their performance, without introducing additional parameters or significantly increasing training overheads into the models. It is also a general framework that can be flexibly extended to distilling various types of knowledge, rather than being restricted to soft labels themselves. Extensive experiments on four benchmarking datasets verify the effectiveness and generality of the proposed approach. The code is available at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/GaoJieCN\/psd\">https:\/\/github.com\/GaoJieCN\/psd<\/jats:ext-link>\n          <\/jats:p>","DOI":"10.1145\/3656168","type":"journal-article","created":{"date-parts":[[2024,4,8]],"date-time":"2024-04-08T12:10:25Z","timestamp":1712578225000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Document-level Relation Extraction with Progressive Self-distillation"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6102-3407","authenticated-orcid":false,"given":"Quan","family":"Wang","sequence":"first","affiliation":[{"name":"MOE Key Laboratory of Trustworthy Distributed Computing and Service, Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5739-8126","authenticated-orcid":false,"given":"Zhendong","family":"Mao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9645-4607","authenticated-orcid":false,"given":"Jie","family":"Gao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1151-1792","authenticated-orcid":false,"given":"Yongdong","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"key":"e_1_3_3_2_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Anil Rohan","year":"2018","unstructured":"Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. 2018. Large scale distributed neural network training through online distillation. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/arxiv.org\/abs\/1804.03235"},{"key":"e_1_3_3_3_2","volume-title":"Advances in Neural Information Processing Systems","author":"Ba Jimmy","year":"2014","unstructured":"Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems, Vol. 27. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2014\/file\/ea8fcd92d59581717e06eb187f10666d-Paper.pdf"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1279"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1371"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1072"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72240-1_21"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3281659"},{"key":"e_1_3_3_9_2","first-page":"345","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Chen Zi-Yuan","year":"2019","unstructured":"Zi-Yuan Chen, Chih-Hung Chang, Yi-Pei Chen, Jijnasa Nayak, and Lun-Wei Ku. 2019. UHop: An unrestricted-hop relation extraction framework for knowledge-based question answering. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 345\u2013356. Retrieved from https:\/\/aclanthology.org\/N19-1031"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.249"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1498"},{"key":"e_1_3_3_12_2","volume-title":"Advances in Neural Information Processing Systems","author":"Czarnecki Wojciech M.","year":"2017","unstructured":"Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. 2017. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/758a06618c69880a6cee5314ee42d52f-Paper.pdf"},{"key":"e_1_3_3_13_2","first-page":"4171","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171\u20134186. Retrieved from https:\/\/aclanthology.org\/N19-1423"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210187"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1026"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988687"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3584701"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531857"},{"key":"e_1_3_3_19_2","first-page":"1607","volume-title":"Proceedings of the 35th International Conference on Machine Learning","volume":"80","author":"Furlanello Tommaso","year":"2018","unstructured":"Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 1607\u20131616. Retrieved from https:\/\/proceedings.mlr.press\/v80\/furlanello18a.html"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.169"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24797-2_4"},{"key":"e_1_3_3_22_2","article-title":"Distilling the knowledge in a neural network","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).","journal-title":"arXiv preprint arXiv:1503.02531"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597610"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.432"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1370"},{"key":"e_1_3_3_26_2","first-page":"1904","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Jiang Feng","year":"2022","unstructured":"Feng Jiang, Jianwei Niu, Shasha Mo, and Shengda Fan. 2022. Key mention pairs guided document-level relation extraction. In Proceedings of the 29th International Conference on Computational Linguistics. 1904\u20131914. Retrieved from https:\/\/aclanthology.org\/2022.coling-1.165"},{"key":"e_1_3_3_27_2","first-page":"1471","volume-title":"Proceedings of the 26th International Conference on Computational Linguistics","author":"Jiang Xiaotian","year":"2016","unstructured":"Xiaotian Jiang, Quan Wang, Peng Li, and Bin Wang. 2016. Relation extraction with multi-instance multi-label convolutional neural networks. In Proceedings of the 26th International Conference on Computational Linguistics. 1471\u20131480. Retrieved from https:\/\/aclanthology.org\/C16-1139"},{"key":"e_1_3_3_28_2","volume-title":"Advances in Neural Information Processing Systems","author":"Kim Jangho","year":"2018","unstructured":"Jangho Kim, Seonguk Park, and Nojun Kwak. 2018. Paraphrasing complex network: Network compression via factor transfer. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2018\/file\/6d9cb7de5e8ac30bd5e8734bc96a35c1-Paper.pdf"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00650"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412011"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1093\/database\/baw068"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.117"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503917"},{"key":"e_1_3_3_34_2","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"e_1_3_3_35_2","article-title":"Decoupled weight decay regularization","author":"Loshchilov Ilya","year":"2017","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).","journal-title":"arXiv preprint arXiv:1711.05101"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.145"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412164"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.5555\/1690219.1690287"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1105"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.141"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.41"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557615"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00409"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186024"},{"key":"e_1_3_3_45_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Romero Adriana","year":"2015","unstructured":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. FitNets: Hints for thin deep nets. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/arxiv.org\/abs\/1412.6550"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2074"},{"key":"e_1_3_3_48_2","first-page":"4723","volume-title":"Proceedings of the 35th International Conference on Machine Learning","volume":"80","author":"Srinivas Suraj","year":"2018","unstructured":"Suraj Srinivas and Francois Fleuret. 2018. Knowledge transfer with Jacobian matching. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 4723\u20134731. Retrieved from https:\/\/proceedings.mlr.press\/v80\/srinivas18a.html"},{"key":"e_1_3_3_49_2","first-page":"455","volume-title":"Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","author":"Surdeanu Mihai","year":"2012","unstructured":"Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 455\u2013465. Retrieved from https:\/\/aclanthology.org\/D12-1042"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.132"},{"key":"e_1_3_3_51_2","first-page":"8472","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Tan Qingyu","year":"2022","unstructured":"Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022. Revisiting DocRED\u2013Addressing the overlooked false negative problem in relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 8472\u20138487. Retrieved from https:\/\/aclanthology.org\/2022.emnlp-main.580"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-47426-3_16"},{"key":"e_1_3_3_53_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Tian Yonglong","year":"2020","unstructured":"Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive representation distillation. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/arxiv.org\/abs\/1910.10699"},{"issue":"86","key":"e_1_3_3_54_2","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten Laurens van der","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 86 (2008), 2579\u20132605. Retrieved from http:\/\/jmlr.org\/papers\/v9\/vandermaaten08a.html","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1080"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2629489"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58548-8_7"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.276"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539304"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-17083-7_17"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.2978386"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.171"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.23"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080768"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17665"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557313"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1074"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.582"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-019-00765-y"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00396"},{"key":"e_1_3_3_71_2","first-page":"2335","volume-title":"Proceedings the 25th International Conference on Computational Linguistics","author":"Zeng Daojian","year":"2014","unstructured":"Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings the 25th International Conference on Computational Linguistics. 2335\u20132344. Retrieved from https:\/\/aclanthology.org\/C14-1220"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.47"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.127"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/551"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00454"},{"key":"e_1_3_3_76_2","first-page":"2184","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang Zhilu","year":"2020","unstructured":"Zhilu Zhang and Mert Sabuncu. 2020. Self-distillation as instance-specific label smoothing. In Advances in Neural Information Processing Systems, Vol. 33. 2184\u20132195. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6509"},{"key":"e_1_3_3_78_2","article-title":"Document-level relation extraction with context guided mention integration and inter-pair reasoning","author":"Zhao Chao","year":"2022","unstructured":"Chao Zhao, Daojian Zeng, Lu Xu, and Jianhua Dai. 2022. Document-level relation extraction with context guided mention integration and inter-pair reasoning. arXiv preprint arXiv:2201.04826 (2022).","journal-title":"arXiv preprint arXiv:2201.04826"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.5"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17717"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2022\/630"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3656168","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3656168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:33Z","timestamp":1750295373000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3656168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,25]]},"references-count":80,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3656168"],"URL":"https:\/\/doi.org\/10.1145\/3656168","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,25]]},"assertion":[{"value":"2023-10-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-27","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}