{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T01:05:50Z","timestamp":1790039150887,"version":"4.0.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2016,4,27]],"date-time":"2016-04-27T00:00:00Z","timestamp":1461715200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2016,4,27]],"date-time":"2016-04-27T00:00:00Z","timestamp":1461715200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Biomed Semant"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Biomedical text mining may target various kinds of valuable information embedded in the literature, but a critical obstacle to the extension of the mining targets is the cost of manual construction of labeled data, which are required for state-of-the-art supervised learning systems. Active learning is to choose the most informative documents for the supervised learning in order to reduce the amount of required manual annotations. Previous works of active learning, however, focused on the tasks of entity recognition and protein-protein interactions, but not on event extraction tasks for multiple event types. They also did not consider the evidence of event participants, which might be a clue for the presence of events in unlabeled documents. Moreover, the confidence scores of events produced by event extraction systems are not reliable for ranking documents in terms of informativity for supervised learning. We here propose a novel committee-based active learning method that supports multi-event extraction tasks and employs a new statistical method for informativity estimation instead of using the confidence scores from event extraction systems.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Our method is based on a committee of two systems as follows: We first employ an event extraction system to filter potential false negatives among unlabeled documents, from which the system does not extract any event. We then develop a statistical method to rank the potential false negatives of unlabeled documents 1) by using a language model that measures the probabilities of the expression of multiple events in documents and 2) by using a named entity recognition system that locates the named entities that can be event arguments (e.g. proteins). The proposed method further deals with unknown words in test data by using word similarity measures. We also apply our active learning method for the task of named entity recognition.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results and conclusion<\/jats:title>\n                    <jats:p>We evaluate the proposed method against the BioNLP Shared Tasks datasets, and show that our method can achieve better performance than such previous methods as entropy and Gibbs error based methods and a conventional committee-based method. We also show that the incorporation of named entity recognition into the active learning for event extraction and the unknown word handling further improve the active learning method. In addition, the adaptation of the active learning method into named entity recognition tasks also improves the document selection for manual annotation of named entities.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s13326-016-0059-z","type":"journal-article","created":{"date-parts":[[2016,4,27]],"date-time":"2016-04-27T12:30:56Z","timestamp":1461760256000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Active learning for ontological event extraction incorporating named entity recognition and unknown word handling"],"prefix":"10.1186","volume":"7","author":[{"given":"Xu","family":"Han","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jung-jae","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chee Keong","family":"Kwoh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2016,4,27]]},"reference":[{"issue":"Suppl 1","key":"59_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-6-S1-S1","volume":"6","author":"L Hirschman","year":"2005","unstructured":"Hirschman L, Yeh A, Blaschke C, Valencia A. Overview of BioCreAtIvE: critical assessment of information extraction for biology. BMC Bioinformatics. 2005; 6(Suppl 1):1.","journal-title":"BMC Bioinformatics"},{"issue":"6","key":"59_CR2","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1093\/bioinformatics\/bti597","volume":"22","author":"J \u0160ari\u0107","year":"2006","unstructured":"\u0160ari\u0107 J, Jensen LJ, Ouzounova R, Rojas I, Bork P. Extraction of regulatory gene\/protein networks from Medline. Bioinformatics. 2006; 22(6):645\u201350.","journal-title":"Bioinformatics"},{"issue":"Suppl 11","key":"59_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-13-S11-S1","volume":"13","author":"JD Kim","year":"2012","unstructured":"Kim JD, Nguyen N, Wang Y, Tsujii J, Takagi T, Yonezawa A. The Genia Event and Protein Coreference tasks of the BioNLP Shared Task 2011. BMC Bioinformatics. 2012; 13(Suppl 11):1.","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"59_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00429ED1V01Y201207AIM018","volume":"6","author":"B Settles","year":"2012","unstructured":"Settles B. Active learning. Synth Lect Artif Intell Mach Learn. 2012; 6(1):1\u2013114.","journal-title":"Synth Lect Artif Intell Mach Learn"},{"issue":"5","key":"59_CR5","doi-asserted-by":"publisher","first-page":"1001","DOI":"10.1136\/amiajnl-2012-001244","volume":"20","author":"Y Chen","year":"2013","unstructured":"Chen Y, Cao H, Mei Q, Zheng K, Xu H. Applying active learning to supervised word sense disambiguation in MEDLINE. J Am Med Inform Assoc. 2013; 20(5):1001\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"59_CR6","volume-title":"Proceedings of the Fifth International Conference on Knowledge Capture","author":"K Tomanek","year":"2009","unstructured":"Tomanek K, Hahn U. Reducing Class Imbalance During Active Learning for Named Entity Annotation. In: Proceedings of the Fifth International Conference on Knowledge Capture. New York, NY, USA: ACM: 2009. p. 105\u201312."},{"key":"59_CR7","doi-asserted-by":"crossref","unstructured":"Tomanek K, Hahn U. Semi-Supervised Active Learning for Sequence Labeling. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: 2009. p. 1039\u201347.","DOI":"10.3115\/1690219.1690291"},{"key":"59_CR8","unstructured":"Tomanek K, Hahn U. A Comparison of Models for Cost-Sensitive Active Learning. In: International Conference on Computational Linguistics (Coling) 2010: Posters: 2010. p. 1247\u20131255."},{"key":"59_CR9","doi-asserted-by":"crossref","unstructured":"Zhang J, Yuan H. A Certainty-Based Active Learning Framework of Meeting Speech Summarization. In: Computer Engineering and Networking: 2014. p. 235\u201342.","DOI":"10.1007\/978-3-319-01766-2_28"},{"key":"59_CR10","doi-asserted-by":"crossref","unstructured":"Lewis DD, Catlett J. Heterogeneous Uncertainty Sampling for Supervised Learning. In: Proceedings of the Eleventh International Conference on Machine Learning: 1994. p. 148\u201356.","DOI":"10.1016\/B978-1-55860-335-6.50026-X"},{"key":"59_CR11","volume-title":"Proceedings of the Fifth Annual Workshop on Computational Learning Theory","author":"HS Seung","year":"1992","unstructured":"Seung HS, Opper M, Sompolinsky H. Query by committee. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory. New York, NY, USA: ACM: 1992. p. 287\u201394."},{"issue":"2","key":"59_CR12","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1007\/s10115-012-0507-8","volume":"35","author":"Y Fu","year":"2013","unstructured":"Fu Y, Zhu X, Li B. A survey on instance selection for active learning. Knowl Inf Syst. 2013; 35(2):249\u201383.","journal-title":"Knowl Inf Syst"},{"key":"59_CR13","volume-title":"Proceedings of the 15th ACM International Conference on Information and Knowledge Management. CIKM \u201906","author":"H Sch\u00fctze","year":"2006","unstructured":"Sch\u00fctze H, Velipasaoglu E, Pedersen JO. Performance Thresholding in Practical Text Classification. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management. CIKM \u201906. New York, NY, USA: ACM: 2006. p. 662\u201371."},{"key":"59_CR14","volume-title":"Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing","author":"K Tomanek","year":"2009","unstructured":"Tomanek K, Laws F, Hahn U, Sch\u00fctze H. On Proper Unit Selection in Active Learning: Co-Selection Effects for Named Entity Recognition. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. Stroudsburg, PA, USA: Association for Computational Linguistics: 2009. p. 9\u201317."},{"key":"59_CR15","volume-title":"Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD \u201910","author":"BC Wallace","year":"2010","unstructured":"Wallace BC, Small K, Brodley CE, Trikalinos TA. Active Learning for Biomedical Citation Screening. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD \u201910. New York, NY, USA: ACM: 2010. p. 173\u201382."},{"issue":"7","key":"59_CR16","doi-asserted-by":"publisher","first-page":"10344","DOI":"10.1016\/j.eswa.2009.01.043","volume":"36","author":"B Cui","year":"2009","unstructured":"Cui B, Lin H, Yang Z. Uncertainty sampling-based active learning for protein-protein interaction extraction from biomedical literature. Expert Syst Appl. 2009; 36(7):10344\u201350.","journal-title":"Expert Syst Appl"},{"issue":"6","key":"59_CR17","doi-asserted-by":"publisher","first-page":"1302","DOI":"10.1007\/s11390-012-1306-0","volume":"27","author":"HT Zhang","year":"2012","unstructured":"Zhang HT, Huang ML, Zhu XY. A unified active learning framework for biomedical relation extraction. J Comput Sci Technol. 2012; 27(6):1302\u201313.","journal-title":"J Comput Sci Technol"},{"key":"59_CR18","volume-title":"Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task","author":"JD Kim","year":"2009","unstructured":"Kim JD, Ohta T, Pyysalo S, Kano Y, Tsujii J. Overview of BioNLP\u201909 Shared Task on Event Extraction. In: Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task. Boulder, Colorado: Association for Computational Linguistics: 2009. p. 1\u20139."},{"key":"59_CR19","volume-title":"Proceedings of the BioNLP Shared Task 2013 Workshop","author":"JJ Kim","year":"2013","unstructured":"Kim JJ, Han X, Lee V, Rebholz-Schuhmann D. GRO Task: Populating the Gene Regulation Ontology with events and relations. In: Proceedings of the BioNLP Shared Task 2013 Workshop. Sofia, Bulgaria: Association for Computational Linguistics: 2013. p. 50\u20137."},{"key":"59_CR20","volume-title":"Proceedings of the BioNLP Shared Task 2013 Workshop","author":"C N\u00e9dellec","year":"2013","unstructured":"N\u00e9dellec C, Bossy R, Kim JD, Kim JJ, Ohta T, Pyysalo S, Zweigenbaum P. Overview of BioNLP Shared Task 2013. In: Proceedings of the BioNLP Shared Task 2013 Workshop. Sofia, Bulgaria: Association for Computational Linguistics: 2013. p. 1\u20137."},{"issue":"4","key":"59_CR21","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1111\/j.1467-8640.2011.00399.x","volume":"27","author":"J Bj\u00f6rne","year":"2011","unstructured":"Bj\u00f6rne J, Heimonen J, Ginter F, Airola A, Pahikkala T, Salakoski T. Extracting Complex Biological Events with Rich Graph-Based Feature Sets. Comput Intell. 2011; 27(4):541\u201357.","journal-title":"Comput Intell"},{"key":"59_CR22","volume-title":"Proceedings of the Fifteenth International Conference on Machine Learning. ICML \u201998","author":"A McCallum","year":"1998","unstructured":"McCallum A, Nigam K. Employing EM and Pool-Based Active Learning for Text Classification. In: Proceedings of the Fifteenth International Conference on Machine Learning. ICML \u201998. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.: 1998. p. 350\u20138."},{"key":"59_CR23","volume-title":"Proceedings of the 20th National Conference on Artificial Intelligence - Volume 2.","author":"A Culotta","year":"2005","unstructured":"Culotta A, McCallum A. Reducing Labeling Effort for Structured Prediction Tasks. In: Proceedings of the 20th National Conference on Artificial Intelligence - Volume 2.Pittsburgh, Pennsylvania: AAAI Press: 2005. p. 746\u201351."},{"key":"59_CR24","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. EMNLP \u201908","author":"B Settles","year":"2008","unstructured":"Settles B, Craven M. An Analysis of Active Learning Strategies for Sequence Labeling Tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. EMNLP \u201908. Stroudsburg, PA, USA: Association for Computational Linguistics: 2008. p. 1070\u20139."},{"issue":"Suppl 11","key":"59_CR25","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/1471-2105-13-S11-S4","volume":"13","author":"J Bj\u00f6rne","year":"2012","unstructured":"Bj\u00f6rne J, Ginter F, Salakoski T. University of Turku in the BioNLP\u201911 shared task. BMC Bioinformatics. 2012; 13(Suppl 11):4.","journal-title":"BMC Bioinformatics"},{"key":"59_CR26","volume-title":"Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics","author":"K Sagae","year":"2007","unstructured":"Sagae K, Miyao Y, Tsujii J. HPSG Parsing with Shallow Dependency Constraints. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Prague, Czech Republic: Association for Computational Linguistics: 2007. p. 624\u201331."},{"key":"59_CR27","doi-asserted-by":"publisher","DOI":"10.1002\/9781118165881","volume-title":"Nonparametric Statistics for Non-statisticians: a Step-by-step Approach","author":"GW Corder","year":"2009","unstructured":"Corder GW, Foreman DI. Nonparametric Statistics for Non-statisticians: a Step-by-step Approach. Hoboken, NJ, USA: John Wiley & Sons, Inc.; 2009."},{"issue":"1","key":"59_CR28","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1186\/1471-2105-14-54","volume":"14","author":"D Campos","year":"2013","unstructured":"Campos D, Matos S, Oliveira JL. Gimli: open source and high-performance biomedical name recognition. BMC Bioinformatics. 2013; 14(1):54.","journal-title":"BMC Bioinformatics"},{"key":"59_CR29","volume-title":"Advances in Neural Information Processing Systems 26","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Advances in Neural Information Processing Systems 26. Red Hook, NY, USA: Curran Associates, Inc.: 2013. p. 3111\u20139."},{"key":"59_CR30","doi-asserted-by":"publisher","first-page":"1651","DOI":"10.1214\/aos\/1024691352","volume":"26","author":"RE Schapire","year":"1998","unstructured":"Schapire RE, Freund Y, Bartlett P, Lee WS. Boosting the margin: A new explanation for the effectiveness of voting methods. Ann Stat. 1998; 26:1651\u201386.","journal-title":"Ann Stat."},{"key":"59_CR31","unstructured":"Vlachos A. Active learning with support vector machines: School of Informatics University of Edinburgh; 2004, pp. 12\u201314."},{"issue":"1","key":"59_CR32","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/584091.584093","volume":"5","author":"CE Shannon","year":"2001","unstructured":"Shannon CE. A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev. 2001; 5(1):3\u201355.","journal-title":"ACM SIGMOBILE Mobile Comput Commun Rev"},{"key":"59_CR33","volume-title":"Advances in Neural Information Processing Systems 26","author":"NV Cuong","year":"2013","unstructured":"Cuong NV, Lee WS, Ye N, Chai KMA, Chieu HL. Active Learning for Probabilistic Hypotheses Using the Maximum Gibbs Error Criterion. In: Advances in Neural Information Processing Systems 26. Red Hook, NY, USA: Curran Associates, Inc.: 2013. p. 1457\u20131465."},{"key":"59_CR34","volume-title":"Proceedings of the BioNLP Shared Task 2013 Workshop","author":"S Pyysalo","year":"2013","unstructured":"Pyysalo S, Ohta T, Ananiadou S. Overview of the Cancer Genetics (CG) task of BioNLP Shared Task 2013. In: Proceedings of the BioNLP Shared Task 2013 Workshop. Sofia, Bulgaria: Association for Computational Linguistics: 2013. p. 58\u201366."},{"key":"59_CR35","volume-title":"Proceedings of the BioNLP Shared Task 2013 Workshop","author":"JD Kim","year":"2013","unstructured":"Kim JD, Wang Y, Yasunori Y. The Genia Event Extraction Shared Task, 2013 Edition - Overview. In: Proceedings of the BioNLP Shared Task 2013 Workshop. Sofia, Bulgaria: Association for Computational Linguistics: 2013. p. 8\u201315."},{"key":"59_CR36","volume-title":"Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. ACL \u201905","author":"JR Finkel","year":"2005","unstructured":"Finkel JR, Grenager T, Manning C. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. ACL \u201905. Stroudsburg, PA, USA: Association for Computational Linguistics: 2005. p. 363\u2013370."},{"key":"59_CR37","volume-title":"Proceedings of CoNLL-2003","author":"EF Tjong Kim Sang","year":"2003","unstructured":"Tjong Kim Sang EF, De Meulder F. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In: Proceedings of CoNLL-2003. Stroudsburg, PA, USA: Association for Computational Linguistics: 2003. p. 142\u20137."},{"issue":"Suppl 2","key":"59_CR38","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/gb-2008-9-s2-s2","volume":"9","author":"L Smith","year":"2008","unstructured":"Smith L, Tanabe L, Ando R, Kuo CJ, Chung IF, Hsu CN, Lin YS, Klinger R, Friedrich C, Ganchev K, Torii M, Liu H, Haddow B, Struble C, Povinelli R, Vlachos A, Baumgartner W, Hunter L, Carpenter B, Tsai R, Dai HJ, Liu F, Chen Y, Sun C, Katrenko S, Adriaans P, Blaschke C, Torres R, Neves M, Nakov P, Divoli A, Mana-Lopez M, Mata J, Wilbur WJ. Overview of BioCreative II gene mention recognition. Genome Biol. 2008; 9(Suppl 2):2.","journal-title":"Genome Biol"},{"key":"59_CR39","doi-asserted-by":"crossref","unstructured":"Zhu J, Wang H, Yao T, Tsou BK. Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification. In: Proceedings of the 22nd International Conference on Computational Linguistics (Manchester, UK: Coling 2008 Organizing Committee): 2008. p. 1137\u20131144.","DOI":"10.3115\/1599081.1599224"},{"key":"59_CR40","first-page":"255","volume":"5","author":"Y Baram","year":"2004","unstructured":"Baram Y, El-Yaniv R, Luz K. Online choice of active learning algorithms. J Mach Learn Res. 2004; 5:255\u201391.","journal-title":"J Mach Learn Res"},{"issue":"10","key":"59_CR41","first-page":"1","volume":"16","author":"JD Kim","year":"2015","unstructured":"Kim JD, Kim J-j, Han X, Rebholz-Schuhmann D. Extending the evaluation of genia event task toward knowledge base construction and comparison to gene regulation ontology task. BMC Bioinformatics. 2015; 16(10):1\u201313.","journal-title":"BMC Bioinformatics"},{"issue":"10","key":"59_CR42","first-page":"1","volume":"16","author":"S Pyysalo","year":"2015","unstructured":"Pyysalo S, Ohta T, Rak R, Rowley A, Chun HW, Jung SJ, Choi SP, Tsujii J, Ananiadou S. Overview of the cancer genetics and pathway curation tasks of bionlp shared task 2013. BMC Bioinformatics. 2015; 16(10):1\u201319.","journal-title":"BMC Bioinformatics"}],"container-title":["Journal of Biomedical Semantics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-016-0059-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13326-016-0059-z","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-016-0059-z","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-016-0059-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T00:28:42Z","timestamp":1790036922000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s13326-016-0059-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,4,27]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2016,12]]}},"alternative-id":["59"],"URL":"https:\/\/doi.org\/10.1186\/s13326-016-0059-z","relation":{},"ISSN":["2041-1480"],"issn-type":[{"value":"2041-1480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,4,27]]},"assertion":[{"value":"21 February 2015","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 March 2016","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 April 2016","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"22"}}