{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,22]],"date-time":"2025-02-22T00:38:36Z","timestamp":1740184716907,"version":"3.37.3"},"reference-count":57,"publisher":"Oxford University Press (OUP)","issue":"23","license":[{"start":{"date-parts":[[2017,7,24]],"date-time":"2017-07-24T00:00:00Z","timestamp":1500854400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/1038099\/1"],"award-info":[{"award-number":["EP\/1038099\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/M006891\/1","BB\/P025684\/1"],"award-info":[{"award-number":["BB\/M006891\/1","BB\/P025684\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["DARPA-BAA-14-14"],"award-info":[{"award-number":["DARPA-BAA-14-14"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>In recent years, there has been great progress in the field of automated curation of biomedical networks and models, aided by text mining methods that provide evidence from literature. Such methods must not only extract snippets of text that relate to model interactions, but also be able to contextualize the evidence and provide additional confidence scores for the interaction in question. Although various approaches calculating confidence scores have focused primarily on the quality of the extracted information, there has been little work on exploring the textual uncertainty conveyed by the author. Despite textual uncertainty being acknowledged in biomedical text mining as an attribute of text mined interactions (events), it is significantly understudied as a means of providing a confidence measure for interactions in pathways or other biomedical models. In this work, we focus on improving identification of textual uncertainty for events and explore how it can be used as an additional measure of confidence for biomedical models.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We present a novel method for extracting uncertainty from the literature using a hybrid approach that combines rule induction and machine learning. Variations of this hybrid approach are then discussed, alongside their advantages and disadvantages. We use subjective logic theory to combine multiple uncertainty values extracted from different sources for the same interaction. Our approach achieves F-scores of 0.76 and 0.88 based on the BioNLP-ST and Genia-MK corpora, respectively, making considerable improvements over previously published work. Moreover, we evaluate our proposed system on pathways related to two different areas, namely leukemia and melanoma cancer research.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>The leukemia pathway model used is available in Pathway Studio while the Ras model is available via PathwayCommons. Online demonstration of the uncertainty extraction system is available for research purposes at http:\/\/argo.nactem.ac.uk\/test. The related code is available on https:\/\/github.com\/c-zrv\/uncertainty_components.git. Details on the above are available in the Supplementary Material.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Supplementary information<\/jats:title>\n                  <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btx466","type":"journal-article","created":{"date-parts":[[2017,7,21]],"date-time":"2017-07-21T19:31:20Z","timestamp":1500665480000},"page":"3784-3792","source":"Crossref","is-referenced-by-count":20,"title":["Using uncertainty to link and rank evidence from biomedical literature for model curation"],"prefix":"10.1093","volume":"33","author":[{"given":"Chrysoula","family":"Zerva","sequence":"first","affiliation":[{"name":"National Centre for Text Mining, School of Computer Science, The University of Manchester, Manchester, UK"}]},{"given":"Riza","family":"Batista-Navarro","sequence":"additional","affiliation":[{"name":"National Centre for Text Mining, School of Computer Science, The University of Manchester, Manchester, UK"}]},{"given":"Philip","family":"Day","sequence":"additional","affiliation":[{"name":"Manchester Institute of Biotechnology, The University of Manchester, Manchester, UK"}]},{"given":"Sophia","family":"Ananiadou","sequence":"additional","affiliation":[{"name":"National Centre for Text Mining, School of Computer Science, The University of Manchester, Manchester, UK"}]}],"member":"286","published-online":{"date-parts":[[2017,7,24]]},"reference":[{"key":"2023020207003361300_btx466-B1","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1093\/bfgp\/elu015","article-title":"Event-based text mining for biology and functional genomics","volume":"14","author":"Ananiadou","year":"2015","journal-title":"Brief. Funct. Genomics"},{"key":"2023020207003361300_btx466-B2","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1038\/nbt924","article-title":"Gaining confidence in high-throughput protein interaction networks","volume":"22","author":"Bader","year":"2004","journal-title":"Nat. Biotechnol"},{"key":"2023020207003361300_btx466-B3","first-page":"183","article-title":"Generalizing biomedical event extraction","volume":"2011","author":"Bj\u00f6rne","year":"2011","journal-title":"Proceedings of the BioNLP"},{"key":"2023020207003361300_btx466-B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-16-S16-S4","article-title":"TEES 2.2: biomedical event extraction for diverse corpora","volume":"16","author":"Bj\u00f6rne","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B5","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1093\/bioinformatics\/btq180","article-title":"Complex event extraction at PubMed scale","volume":"26","author":"Bj\u00f6rne","year":"2010","journal-title":"Bioinformatics"},{"key":"2023020207003361300_btx466-B6","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1145\/253262.253325","article-title":"Dynamic itemset counting and implication rules for market basket data","author":"Brin","year":"1997","journal-title":"Proceedings of ACM SIGMOD International Conference on Management of Data"},{"key":"2023020207003361300_btx466-B7","doi-asserted-by":"crossref","first-page":"045008","DOI":"10.1088\/1478-3975\/12\/4\/045008","article-title":"Darpa\u2019s big mechanism program","volume":"12","author":"Cohen","year":"2015","journal-title":"Phys. Biol"},{"key":"2023020207003361300_btx466-B8","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1007\/978-1-4939-0709-0_8","article-title":"Mining biological networks from full-text articles","volume":"1159","author":"Czarnecki","year":"2014","journal-title":"Methods Mol. Biol"},{"key":"2023020207003361300_btx466-B9","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1186\/1471-2105-4-11","article-title":"Prebind and textomy\u2013mining the biomedical literature for protein-protein interactions using a support vector machine","volume":"4","author":"Donaldson","year":"2003","journal-title":"BMC Bioinformatics"},{"year":"2010","author":"Farkas","key":"2023020207003361300_btx466-B10"},{"key":"2023020207003361300_btx466-B11","first-page":"1","article-title":"Text mining for metabolic pathways, signaling cascades, and protein networks","volume":"283","author":"Hoffmann","year":"2005","journal-title":"Sci. STKE"},{"key":"2023020207003361300_btx466-B12","first-page":"199","article-title":"Assessment of biomedical knowledge according to confidence criteria","volume":"136","author":"Jilani","year":"2008","journal-title":"Stud. Health Technol. Inform"},{"key":"2023020207003361300_btx466-B13","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1142\/S0218488501000831","article-title":"A Logic for Uncertain Probabilities","volume":"vol. 9","author":"J\u00f8sang","year":"2001","journal-title":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems"},{"year":"2006","author":"J\u00f8sang","key":"2023020207003361300_btx466-B14"},{"year":"2017","author":"J\u00f8sang","key":"2023020207003361300_btx466-B15"},{"key":"2023020207003361300_btx466-B16","first-page":"22","article-title":"A compositional interpretation of biomedical event factuality","volume":"2015","author":"Kilicoglu","year":"2015","journal-title":"ExProM"},{"key":"2023020207003361300_btx466-B17","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1093\/bioinformatics\/btg1023","article-title":"GENIA corpus-a semantically annotated corpus for bio-textmining","volume":"19","author":"Kim","year":"2003","journal-title":"Bioinformatics"},{"key":"2023020207003361300_btx466-B18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5465\/ambpp.2009.44256545","article-title":"Overview of BioNLP\u201909 shared task on event extraction","volume":"2009","author":"Kim","year":"2009","journal-title":"Proceedings of BioNLP"},{"key":"2023020207003361300_btx466-B19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5465\/ambpp.2011.1.1fy","article-title":"Overview of BioNLP shared task 2011","volume":"2011","author":"Kim","year":"2011","journal-title":"Proceedings of BioNLP"},{"key":"2023020207003361300_btx466-B20","first-page":"18","article-title":"Classification and regression by randomForest","volume":"2","author":"Liaw","year":"2002","journal-title":"R News"},{"key":"2023020207003361300_btx466-B21","first-page":"17","article-title":"The language of bioscience: Facts, speculations, and statements in between","volume":"2004","author":"Light","year":"2004","journal-title":"Proceedings of BioLink"},{"key":"2023020207003361300_btx466-B22","doi-asserted-by":"crossref","first-page":"100\u2013117.","DOI":"10.1371\/journal.pcbi.1003117","article-title":"Hypothesis Finder: a Strategy for the Detection of Speculative Statements in Scientific Text","volume":"9","author":"Malhotra","year":"2013","journal-title":"PLoS Comput. Biol"},{"key":"2023020207003361300_btx466-B23","first-page":"545","article-title":"Comparative parser performance analysis across grammar frameworks through automatic tree conversion using synchronous grammars","volume":"1","author":"Matsuzaki","year":"2008","journal-title":"Proceedings of the 22nd ACL"},{"key":"2023020207003361300_btx466-B24","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1016\/j.jbi.2008.01.001","article-title":"Exploring hedge identification in biomedical literature","volume":"41","author":"Medlock","year":"2008","journal-title":"J. Biomed. Informatics"},{"key":"2023020207003361300_btx466-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-16-S10-S7","article-title":"Adaptable, high recall, event extraction system with minimal configuration","volume":"16","author":"Miwa","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B26","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1186\/1471-2105-13-108","article-title":"Extracting semantically enriched events from biomedical literature","volume":"29","author":"Miwa","year":"2012","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B27","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1093\/bioinformatics\/btt227","article-title":"A method for integrating and ranking the evidence for biochemical pathways by mining reactions from text","volume":"29","author":"Miwa","year":"2013","journal-title":"Bioinformatics"},{"first-page":"31","year":"2014","author":"Mowery","key":"2023020207003361300_btx466-B28"},{"key":"2023020207003361300_btx466-B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-14-14","article-title":"Negated bio-events: analysis and identification","volume":"14","author":"Nawaz","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B30","first-page":"1","article-title":"Overview of BioNLP shared task 2013","author":"N\u00e9dellec","year":"2013","journal-title":"Proceedings of BioNLP"},{"key":"2023020207003361300_btx466-B31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-9-S3-S5","article-title":"New challenges for text mining: mapping between text and manually curated pathways","volume":"9","author":"Oda","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B32","doi-asserted-by":"crossref","first-page":"12.","DOI":"10.1186\/1756-0381-1-12","article-title":"A survey of visualization tools for biological network analysis","volume":"1","author":"Pavlopoulos","year":"2008","journal-title":"Biodata Mining"},{"key":"2023020207003361300_btx466-B33","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1016\/j.tips.2009.11.006","article-title":"Unveiling the role of network and systems biology in drug discovery","volume":"31","author":"Pujol","year":"2010","journal-title":"Trends Pharmacol. Sci"},{"key":"2023020207003361300_btx466-B34","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1093\/bioinformatics\/bts407","article-title":"Event extraction across multiple levels of biological organization","volume":"28","author":"Pyysalo","year":"2012","journal-title":"Bioinformatics"},{"key":"2023020207003361300_btx466-B35","first-page":"141","article-title":"Stating with certainty or stating with doubt: Intercoder reliability results for manual annotation of epistemically modalized statements","author":"Rubin","year":"2007","journal-title":"Human Language Technologies 2007: NAACL"},{"key":"2023020207003361300_btx466-B36","first-page":"38","article-title":"Toward fine-grained annotation of modality in text","author":"Rubinstein","year":"2013","journal-title":"Proceedings of IWCS 2013 WAMM"},{"key":"2023020207003361300_btx466-B37","doi-asserted-by":"crossref","first-page":"e1000411","DOI":"10.1371\/journal.pcbi.1000411","article-title":"Getting started in text mining: part two","volume":"5","author":"Rzhetsky","year":"2009","journal-title":"PLoS Comput Biol"},{"key":"2023020207003361300_btx466-B38","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1093\/bioinformatics\/bti597","article-title":"Extraction of regulatory gene\/protein networks from medline","volume":"22","author":"\u0160ari\u0107","year":"2005","journal-title":"Bioinformatics"},{"key":"2023020207003361300_btx466-B39","doi-asserted-by":"crossref","first-page":"e31826.","DOI":"10.1371\/journal.pone.0031826","article-title":"Hippie: Integrating protein interaction networks with experiment based quality scores","volume":"7","author":"Schaefer","year":"2012","journal-title":"PloS One"},{"key":"2023020207003361300_btx466-B40","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1089\/106652703322756104","article-title":"Mining the biomedical literature in the genomic era: an overview","volume":"10","author":"Shatkay","year":"2003","journal-title":"J. Comput. Biol"},{"key":"2023020207003361300_btx466-B41","doi-asserted-by":"crossref","first-page":"1), 17.","DOI":"10.1186\/s13040-016-0096-2","article-title":"Building a glaucoma interaction network using a text mining approach","volume":"9","author":"Soliman","year":"2016","journal-title":"BioData Mining"},{"key":"2023020207003361300_btx466-B42","first-page":"102","article-title":"Brat: a web-based tool for nlp-assisted text annotation","author":"Stenetorp","year":"2012","journal-title":"Proceedings of Demonstrations at 13th EACL"},{"year":"2012","author":"Stenetorp","key":"2023020207003361300_btx466-B43"},{"key":"2023020207003361300_btx466-B44","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.jbi.2015.01.006","article-title":"HPIminer: A text mining system for building and visualizing human protein interaction networks and pathways","volume":"54","author":"Subramani","year":"2015","journal-title":"J. Biomed. Inform"},{"key":"2023020207003361300_btx466-B45","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1162\/COLI_a_00098","article-title":"Cross-genre and cross-domain detection of semantic uncertainty","volume":"38","author":"Szarvas","year":"2012","journal-title":"Comput. Linguist"},{"key":"2023020207003361300_btx466-B46","doi-asserted-by":"crossref","first-page":"D561","DOI":"10.1093\/nar\/gkq973","article-title":"The string database in 2011: functional interaction networks of proteins, globally integrated and scored","volume":"39(Suppl. 1)","author":"Szklarczyk","year":"2011","journal-title":"Nucleic Acids Res"},{"year":"2010","author":"Tang","key":"2023020207003361300_btx466-B47"},{"key":"2023020207003361300_btx466-B48","doi-asserted-by":"crossref","first-page":"349.","DOI":"10.1186\/1471-2105-10-349","article-title":"Construction of an annotated corpus to support biomedical information extraction","volume":"10","author":"Thompson","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B49","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1186\/1471-2105-12-393","article-title":"Enriching a biomedical event corpus with meta-knowledge annotation","volume":"12","author":"Thompson","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2023020207003361300_btx466-B50","first-page":"1","article-title":"Enriching news events with meta-knowledge information","volume":"51","author":"Thompson","year":"2016","journal-title":"LREC"},{"key":"2023020207003361300_btx466-B51","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1093\/bioinformatics\/bti187","article-title":"An architecture for biological information extraction and representation","volume":"21","author":"Vailaya","year":"2005","journal-title":"Bioinformatics"},{"key":"2023020207003361300_btx466-B52","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pone.0055814","article-title":"Large-Scale Event Extraction from Literature with Multi-Level Gene Normalization","volume":"8","author":"Van Landeghem","year":"2013","journal-title":"PLoS One"},{"key":"2023020207003361300_btx466-B53","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1162\/COLI_a_00126","article-title":"Speculation and negation: Rules, rankers, and the role of syntax","volume":"38","author":"Velldal","year":"2012","journal-title":"Comput. Linguist"},{"key":"2023020207003361300_btx466-B54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-9-S11-S9","article-title":"The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes","volume":"9","author":"Vincze","year":"2008","journal-title":"BMC Bioinformatics"},{"year":"2015","author":"Xu","key":"2023020207003361300_btx466-B55"},{"key":"2023020207003361300_btx466-B56","doi-asserted-by":"crossref","first-page":"e0133715","DOI":"10.1371\/journal.pone.0133715","article-title":"Hedge scope detection in biomedical texts: an effective dependency-based method","volume":"10","author":"Zhou","year":"2015","journal-title":"PLOS One"},{"key":"2023020207003361300_btx466-B57","first-page":"968","article-title":"Tree Kernel-based negation and speculation scope detection with structured syntactic Parse Features","volume":"2013","author":"Zou","year":"2013","journal-title":"Proceedings of EMNLP"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/23\/3784\/49041861\/bioinformatics_33_23_3784.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/23\/3784\/49041861\/bioinformatics_33_23_3784.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T07:01:42Z","timestamp":1675321302000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/33\/23\/3784\/4004870"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2017,7,24]]},"references-count":57,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2017,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btx466","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"type":"print","value":"1367-4803"},{"type":"electronic","value":"1367-4811"}],"subject":[],"published-other":{"date-parts":[[2017,12,1]]},"published":{"date-parts":[[2017,7,24]]}}}