{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:19:22Z","timestamp":1760242762807,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2016,5,24]],"date-time":"2016-05-24T00:00:00Z","timestamp":1464048000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National 863 Program of China","award":["2015AA015405"],"award-info":[{"award-number":["2015AA015405"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61402128","61473101","61272383"],"award-info":[{"award-number":["61402128","61473101","61272383"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Strategic Emerging Industry Development Special Funds of Shenzhen","award":["JCYJ20140417172417105","JCYJ20140508161040764","JCYJ20140627163809422"],"award-info":[{"award-number":["JCYJ20140417172417105","JCYJ20140508161040764","JCYJ20140627163809422"]}]},{"name":"Innovation Fund of Harbin Institute of Technology","award":["HIT.NSRIF.2017052"],"award-info":[{"award-number":["HIT.NSRIF.2017052"]}]},{"name":"Program from the Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education","award":["93K172016K12"],"award-info":[{"award-number":["93K172016K12"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Relation extraction has benefited from distant supervision in recent years with the development of natural language processing techniques and data explosion. However, distant supervision is still greatly limited by the quality of training data, due to its natural motivation for greatly reducing the heavy cost of data annotation. In this paper, we construct an architecture called MIML-sort (Multi-instance Multi-label Learning with Sorting Strategies), which is built on the famous MIML framework. Based on MIML-sort, we propose three ranking-based methods for sample selection with which we identify relation extractors from a subset of the training data. Experiments are set up on the KBP (Knowledge Base Propagation) corpus, one of the benchmark datasets for distant supervision, which is large and noisy. Compared with previous work, the proposed methods produce considerably better results. Furthermore, the three methods together achieve the best F1 on the official testing set, with an optimal enhancement of F1 from 27.3% to 29.98%.<\/jats:p>","DOI":"10.3390\/e18060204","type":"journal-article","created":{"date-parts":[[2016,5,24]],"date-time":"2016-05-24T21:53:40Z","timestamp":1464126820000},"page":"204","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Distant Supervision for Relation Extraction with Ranking-Based Methods"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1395-6805","authenticated-orcid":false,"given":"Yang","family":"Xiang","sequence":"first","affiliation":[{"name":"Intelligence Computing Research Center, Harbin Institute of Technology Shenzhen Graduate School, Shenzhen 518055, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qingcai","family":"Chen","sequence":"additional","affiliation":[{"name":"Intelligence Computing Research Center, Harbin Institute of Technology Shenzhen Graduate School, Shenzhen 518055, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaolong","family":"Wang","sequence":"additional","affiliation":[{"name":"Intelligence Computing Research Center, Harbin Institute of Technology Shenzhen Graduate School, Shenzhen 518055, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Qin","sequence":"additional","affiliation":[{"name":"Intelligence Computing Research Center, Harbin Institute of Technology Shenzhen Graduate School, Shenzhen 518055, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,5,24]]},"reference":[{"key":"ref_1","unstructured":"Miller, S., Fox, H., Ramshaw, L., and Weischedel, R. (2000, January 29). A Novel Use of Statistical Parsing to Extract Information from Text. Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference (NAACL), San Diego, CA, USA."},{"key":"ref_2","unstructured":"Collins, M., and Duffy, N. (2001, January 3\u20138). Convolution Kernels for Natural Language. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada."},{"key":"ref_3","first-page":"1083","article-title":"Kernel Methods for Relation Extraction","volume":"3","author":"Zelenko","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kambhatla, N. (2004, January 21\u201326). Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain.","DOI":"10.3115\/1219044.1219066"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Culotta, A., and Sorensen, J. (2004, January 21\u201326). Dependency Tree Kernels for Relation Extraction. Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain.","DOI":"10.3115\/1218955.1219009"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bunescu, R.C., and Mooney, R.J. (2005, January 6\u20138). A Shortest Path Dependency Kernel for Relation Extraction. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT\/EMNLP), Vancouver, BC, Canada.","DOI":"10.3115\/1220575.1220666"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhao, S., and Grishman, R. (2005, January 25\u201330). Extracting Relations with Integrated Information Using Kernel Methods. Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), Ann Harbor, MI, USA.","DOI":"10.3115\/1219840.1219892"},{"key":"ref_8","unstructured":"Zhou, G., Su, J., Zhang, J., and Zhang, M. (2005, January 25\u201330). Exploring Various Knowledge in Relation Extraction. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL), Ann Harbor, MI, USA."},{"key":"ref_9","unstructured":"Bach, N., and Badaskar, S. A Review of Relation Extraction. Available online: orb.essex.ac.uk\/CE\/CE807\/Readings\/A-survey-on-Relation-Extraction.pdf."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mintz, M., Bills, S., Snow, R., and Jurafsky, D. (2009, January 2\u20137). Distant Supervision for Relation Extraction without Labeled Data. Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Singapore.","DOI":"10.3115\/1690219.1690287"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Riedel, S., Yao, L., and McCallum, A. (2010, January 20\u201324). Modeling Relations and Their Mentions without Labeled Text. Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, Barcelona, Spain.","DOI":"10.1007\/978-3-642-15939-8_10"},{"key":"ref_12","unstructured":"Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., and Weld, D.S. (2011, January 19\u201324). Knowledge Based Weak Supervision for Information Extraction of Overlapping Relations. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), Portland, OR, USA."},{"key":"ref_13","unstructured":"Surdeanu, M., Tibshirani, J., Nallapati, R., and Manning, C.D. (2012, January 12\u201314). Multi-Instance Multi-Label Learning for Relation Extraction. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP), Jeju Island, Korea."},{"key":"ref_14","unstructured":"Min, B., Grishman, R., Wan, L., Wang, C., and Gondek, D. (2013, January 9\u201315). Distant Supervision for Relation Extraction with an Incomplete Knowledge Base. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Atlanta, GA, USA."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1162\/tacl_a_00234","article-title":"Modeling Missing Data in Distant Supervision for Information Extraction","volume":"1","author":"Ritter","year":"2013","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_16","unstructured":"Craven, M., and Kumlien, J. (1999, January 6\u201310). Constructing Biological Knowledge Bases by Extracting Information from Text Sources. Proceedings of the 7th International Conference on Intelligent Systems for Molecular Biology (ISMB), Heidelberg, Germany."},{"key":"ref_17","unstructured":"Bunescu, R.C., and Mooney, R.J. (2007, January 23\u201330). Learning to Extract Relations from the Web Using Minimal Supervision. Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), Prague, Czech."},{"key":"ref_18","unstructured":"Bellare, K., and McCallum, A. Learning Extractors from Unlabeled Text Using Relevant Databases. Available online: http:\/\/www.aaai.org\/Papers\/Workshops\/2007\/WS-07-14\/WS07-14-002.pdf."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wu, F., and Weld, D. (2007, January 6\u201310). Autonomously Semantifying Wikipedia. Proceedings of the 16th International Conference on Information and Knowledge Management (CIKM), Lisbon, Portugal.","DOI":"10.1145\/1321440.1321449"},{"key":"ref_20","unstructured":"Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013, January 5\u201310). Translating Embeddings for Modeling Multi-relational Data. Proceedings of the 26th Advances in Neural Information Processing Systems (NIPS), South Lake Tahoe, NV, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, Z., Zhang, J., Feng, J., and Chen, Z. (2014, January 27\u201331). Knowledge Graph Embedding by Translating on Hyperplanes. Proceedings of the 29th AAAI Conference on Artificial Intelligence, Quebec City, QC, Canada.","DOI":"10.1609\/aaai.v28i1.8870"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lin, Y., Liu, Z., Sun, M., Liu, Y., and Zhu, X. (2015, January 25\u201330). Learning Entity and Relation Embeddings for Knowledge Graph Completion. Proceedings of the 30th AAAI Conference on Artificial Intelligence, Austin, TX, USA.","DOI":"10.1609\/aaai.v29i1.9491"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Fan, M., Zhao, D., Zhou, Q., Liu, Z., Zheng, T.F., and Chang, E.Y. (2014, January 22\u201327). Distant Supervision for Relation Extraction with Matrix Completion. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-1079"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Angeli, G., Tibshirani, J., Wu, J.Y., and Manning, C.D. (2014, January 25\u201329). Combining Distant and Partial Supervision for Relation Extraction. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1164"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Nagesh, A., Haffari, G., and Ramakrishna, G. (2014, January 25\u201329). Noisy-or Based Model for Relation Extraction Using Distant Supervision. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1208"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zeng, D., Liu, K., Chen, Y., and Zhao, J. (2015, January 17\u201321). Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal.","DOI":"10.18653\/v1\/D15-1203"},{"key":"ref_27","first-page":"41","article-title":"Removing Noisy Mentions for Distant Supervision","volume":"51","author":"Intxaurrondo","year":"2013","journal-title":"Proces. Leng. Nat."},{"key":"ref_28","unstructured":"Xu, W., Hoffmann, R., Zhao, L., and Grishman, R. (2013, January 4\u20139). Filling Knowledge Base Gaps for Distant Supervision of Relation Extraction. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), Sofia, Bulgaria."},{"key":"ref_29","unstructured":"Takamatsu, S., Sato, I., and Nakagawa, H. (2012, January 8\u201314). Reducing Wrong Labels in Distant Supervision for Relation Extraction. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), Jeju Island, Korea."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"969053","DOI":"10.1155\/2015\/969053","article-title":"Bias Modeling for Distantly Supervised Relation Extraction","volume":"2015","author":"Xiang","year":"2015","journal-title":"Math. Probl. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Xiang, Y., Wang, X., Zhang, Y., Qin, Y., and Fan, S. (2015, January 9\u201312). Distant Supervision for Relation Extraction via Group Selection. Proceedings of the 22nd International Conference on Neural Information Processing (ICONIP), Istanbul, Turkey.","DOI":"10.1007\/978-3-319-26535-3_29"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Usunier, N., Buffoni, D., and Gallinari, P. (2009, January 14\u201318). Ranking with Ordered Weighted Pairwise Classification. Proceedings of the 26th International Conference on Machine Learning (ICML), Montreal, QC, Canada.","DOI":"10.1145\/1553374.1553509"},{"key":"ref_33","unstructured":"Weston, J., Bengio, S., and Usunier, N. (2011, January 16\u201322). Wsabie: Scaling up to Large Vocabulary Image Annotation. Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI), Barcelona, Spain."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Huang, S., Gao, W., and Zhou, Z.H. (2014, January 27\u201331). Fast Multi-Instance Multi-Label Learning. Proceedings of the 2014 AAAI Conference on Artificial Intelligence (AAAI), Hilton, QC, Canada.","DOI":"10.1609\/aaai.v28i1.8970"},{"key":"ref_35","unstructured":"Ji, H., Grishman, R., Dang, H.T., Griffitt, K., and Ellis, J. (2010, January 15\u201316). Overview of the TAC 2010 Knowledge Base Population Track. Proceedings of the Third Text Analysis Conference (TAC 2010), Gaithersburg, MD, USA."},{"key":"ref_36","unstructured":"Ji, H., Grishman, R., and Dang, H.T. (2011, January 14\u201315). Overview of the TAC 2011 Knowledge Base Population Track. Proceedings of the Forth Text Analytics Conference (TAC 2011), Gaithersburg, MD, USA."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/6\/204\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:24:24Z","timestamp":1760210664000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/6\/204"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,5,24]]},"references-count":36,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2016,6]]}},"alternative-id":["e18060204"],"URL":"https:\/\/doi.org\/10.3390\/e18060204","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2016,5,24]]}}}