{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T11:08:36Z","timestamp":1773659316509,"version":"3.50.1"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"S1","license":[{"start":{"date-parts":[[2013,4,1]],"date-time":"2013-04-01T00:00:00Z","timestamp":1364774400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/2.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2013,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Background<\/jats:title>\n            <jats:p>Most previous Protein Protein Interaction (PPI) studies evaluated their algorithms' performance based on \"per-instance\" precision and recall, in which the instances of an interaction relation were evaluated independently. However, we argue that this standard evaluation method should be revisited. In a large corpus, the same relation can be described in various different forms and, in practice, correctly identifying not all but a small subset of them would often suffice to detect the given interaction.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>In this regard, we propose a more pragmatic \"per-relation\" basis performance evaluation method instead of the conventional per-instance basis method. In the per-relation basis method, only a subset of a relation's instances needs to be correctly identified to make the relation positive. In this work, we also introduce a new high-precision rule-based PPI extraction algorithm. While virtually all current PPI extraction studies focus on improving F-score, aiming to balance the performance on both precision and recall, in many realistic scenarios involving large corpora, one can benefit more from a high-precision algorithm than a high-recall counterpart.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>We show that our algorithm not only achieves better per-relation performance than previous solutions but also serves as a good complement to the existing PPI extraction tools. Our algorithm improves the performance of the existing tools through simple pipelining.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>The significance of this research can be found in that this research brought new perspective to the performance evaluation of PPI extraction studies, which we believe is more important in practice than existing evaluation criteria. Given the new evaluation perspective, we also showed the importance of a high-precision extraction tool and validated the efficacy of our rule-based system as the high-precision tool candidate.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1186\/1472-6947-13-s1-s7","type":"journal-article","created":{"date-parts":[[2019,12,11]],"date-time":"2019-12-11T02:01:34Z","timestamp":1576029694000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["On the efficacy of per-relation basis performance evaluation for PPI extraction and a high-precision rule-based approach"],"prefix":"10.1186","volume":"13","author":[{"given":"Junkyu","family":"Lee","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seongsoon","family":"Kim","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sunwon","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kyubum","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jaewoo","family":"Kang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2013,4,5]]},"reference":[{"key":"654_CR1","doi-asserted-by":"publisher","first-page":"54","DOI":"10.3115\/1641484.1641492","volume-title":"Proceedings of the ACL-ISMB Workshop on Linking Biological Literature, Ontologies and Databases: Mining Biological Semantics","author":"S Ahmed","year":"2005","unstructured":"Ahmed S, Chidambaram D, Davulcu H, Baral C: Intex: A syntactic role driven protein-protein interaction extractor for bio-medical text. Proceedings of the ACL-ISMB Workshop on Linking Biological Literature, Ontologies and Databases: Mining Biological Semantics. 2005, Association for Computational Linguistics, 54-61."},{"issue":"3","key":"654_CR2","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1093\/bioinformatics\/btl616","volume":"23","author":"K Fundel","year":"2007","unstructured":"Fundel K, K\u00fcffner R, Zimmer R: RelEx-Relation extraction using dependency parse trees. Bioinformatics. 2007, 23 (3): 365-371. 10.1093\/bioinformatics\/btl616.","journal-title":"Bioinformatics"},{"issue":"3","key":"654_CR3","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/S1532-0464(03)00039-X","volume":"36","author":"G Leroy","year":"2003","unstructured":"Leroy G, Chen H, Martinez J: A shallow parser based on closed-class words to capture relations in biomedical text. Journal of biomedical Informatics. 2003, 36 (3): 145-158. 10.1016\/S1532-0464(03)00039-X.","journal-title":"Journal of biomedical Informatics"},{"issue":"2","key":"654_CR4","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/j.artmed.2006.08.005","volume":"39","author":"F Rinaldi","year":"2007","unstructured":"Rinaldi F, Schneider G, Kaljurand K, Hess M, Andronis C, Konstandi O, Persidis A: Mining of relations between proteins over biomedical scientific literature using a deep-linguistic approach. Artificial intelligence in medicine. 2007, 39 (2): 127-136. 10.1016\/j.artmed.2006.08.005.","journal-title":"Artificial intelligence in medicine"},{"issue":"Suppl 11","key":"654_CR5","doi-asserted-by":"publisher","first-page":"S2","DOI":"10.1186\/1471-2105-9-S11-S2","volume":"9","author":"A Airola","year":"2008","unstructured":"Airola A, Pyysalo S, Bj\u00f6rne J, Pahikkala T, Ginter F, Salakoski T: All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning. BMC bioinformatics. 2008, 9 (Suppl 11): S2-10.1186\/1471-2105-9-S11-S2.","journal-title":"BMC bioinformatics"},{"issue":"2","key":"654_CR6","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1093\/bioinformatics\/btq620","volume":"27","author":"Q Bui","year":"2011","unstructured":"Bui Q, Katrenko S, Sloot P: A hybrid approach to extract protein-protein interactions. Bioinformatics. 2011, 27 (2): 259-265. 10.1093\/bioinformatics\/btq620.","journal-title":"Bioinformatics"},{"issue":"12","key":"654_CR7","doi-asserted-by":"publisher","first-page":"e39","DOI":"10.1016\/j.ijmedinf.2009.04.010","volume":"78","author":"M Miwa","year":"2009","unstructured":"Miwa M, S\u00e6tre R, Miyao Y, Tsujii J: Protein-protein interaction extraction by leveraging multiple kernels and parsers. International Journal of Medical Informatics. 2009, 78 (12): e39-e46. 10.1016\/j.ijmedinf.2009.04.010.","journal-title":"International Journal of Medical Informatics"},{"issue":"4","key":"654_CR8","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1109\/TCBB.2012.50","volume":"9","author":"Y Zhang","year":"2012","unstructured":"Zhang Y, Lin H, Yang Z, Wang J, Li Y: Hash Subgraph Pairwise Kernel for Protein-Protein Interaction Extraction. IEEE\/ACM Transactions on Computational Biology and Bioinformatics (TCBB). 2012, 9 (4): 1190-1202.","journal-title":"IEEE\/ACM Transactions on Computational Biology and Bioinformatics (TCBB)"},{"key":"654_CR9","doi-asserted-by":"publisher","first-page":"724","DOI":"10.3115\/1220575.1220666","volume-title":"Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing","author":"R Bunescu","year":"2005","unstructured":"Bunescu R, Mooney R: A shortest path dependency kernel for relation extraction. Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing. 2005, Association for Computational Linguistics, 724-731."},{"key":"654_CR10","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1145\/2390068.2390082","volume-title":"Proceedings of the ACM Sixth International Workshop on Data and Text Mining in Biomedical Informatics","author":"J Lee","year":"2012","unstructured":"Lee J, Kim S, Lee S, Lee K, Kang J: High precision rule based PPI extraction and per-pair basis performance evaluation. Proceedings of the ACM Sixth International Workshop on Data and Text Mining in Biomedical Informatics. 2012, New York: ACM, 69-76. 10.1145\/2390068.2390082."},{"key":"654_CR11","first-page":"60","volume-title":"Proc Int Conf Intell Syst Mol Biol, Volume 7","author":"C Blaschke","year":"1999","unstructured":"Blaschke C, Andrade M, Ouzounis C, Valencia A: Automatic extraction of biological information from scientific text: protein-protein interactions. Proc Int Conf Intell Syst Mol Biol, Volume 7. 1999, 60-67."},{"issue":"18","key":"654_CR12","doi-asserted-by":"publisher","first-page":"3604","DOI":"10.1093\/bioinformatics\/bth451","volume":"20","author":"M Huang","year":"2004","unstructured":"Huang M, Zhu X, Hao Y, Payan D, Qu K, Li M: Discovering patterns to extract protein-protein interactions from full texts. Bioinformatics. 2004, 20 (18): 3604-3612. 10.1093\/bioinformatics\/bth451.","journal-title":"Bioinformatics"},{"issue":"2","key":"654_CR13","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1093\/bioinformatics\/17.2.155","volume":"17","author":"T Ono","year":"2001","unstructured":"Ono T, Hishigaki H, Tanigami A, Takagi T: Automated extraction of information on protein-protein interactions from the biological literature. Bioinformatics. 2001, 17 (2): 155-161. 10.1093\/bioinformatics\/17.2.155.","journal-title":"Bioinformatics"},{"key":"654_CR14","first-page":"757","volume-title":"Proceedings of the 23rd International Conference on Computational Linguistics: Posters","author":"B Liu","year":"2010","unstructured":"Liu B, Qian L, Wang H, Zhou G: Dependency-driven feature-based learning for extracting protein-protein interactions from biomedical text. Proceedings of the 23rd International Conference on Computational Linguistics: Posters. 2010, Association for Computational Linguistics, 757-765."},{"key":"654_CR15","doi-asserted-by":"publisher","first-page":"121","DOI":"10.3115\/1699510.1699527","volume-title":"Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1","author":"M Miwa","year":"2009","unstructured":"Miwa M, S\u00e6tre R, Miyao Y, Tsujii J: A rich feature vector for protein-protein interaction extraction from multiple corpora. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. 2009, Association for Computational Linguistics, 121-130."},{"key":"654_CR16","volume-title":"Proceedings of the International Symposium on Languages in Biology and Medicine (LBM short oral presentations)","author":"R S\u00e6tre","year":"2007","unstructured":"S\u00e6tre R, Sagae K, Tsujii J: Syntactic features for protein-protein interaction extraction. Proceedings of the International Symposium on Languages in Biology and Medicine (LBM short oral presentations). 2007, Citeseer"},{"key":"654_CR17","first-page":"625","volume-title":"Proceedings of NIPS, Volume 14","author":"M Collins","year":"2001","unstructured":"Collins M, Duffy N: Convolution kernels for natural language. Proceedings of NIPS, Volume 14. 2001, 625-632."},{"key":"654_CR18","first-page":"171","volume":"18","author":"R Bunescu","year":"2006","unstructured":"Bunescu R, Mooney R: Subsequence kernels for relation extraction. Advances in neural information processing systems. 2006, 18: 171-","journal-title":"Advances in neural information processing systems"},{"key":"654_CR19","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1186\/1471-2105-11-107","volume":"11","author":"S Kim","year":"2010","unstructured":"Kim S, Yoon J, Yang J, Park S: Walk-weighted subsequence kernels for protein-protein interaction extraction. BMC bioinformatics. 2010, 11: 107-10.1186\/1471-2105-11-107.","journal-title":"BMC bioinformatics"},{"key":"654_CR20","first-page":"206","volume-title":"Proceedings of the 23rd International Conference on Computational Linguistics","author":"S Choi","year":"2010","unstructured":"Choi S, Myaeng S: Simplicity is better: revisiting single kernel PPI extraction. Proceedings of the 23rd International Conference on Computational Linguistics. 2010, Association for Computational Linguistics, 206-214."},{"key":"654_CR21","first-page":"1083","volume":"3","author":"D Zelenko","year":"2003","unstructured":"Zelenko D, Aone C, Richardella A: Kernel methods for relation extraction. The Journal of Machine Learning Research. 2003, 3: 1083-1106.","journal-title":"The Journal of Machine Learning Research"},{"issue":"7","key":"654_CR22","doi-asserted-by":"publisher","first-page":"e1000837","DOI":"10.1371\/journal.pcbi.1000837","volume":"6","author":"D Tikk","year":"2010","unstructured":"Tikk D, Thomas P, Palaga P, Hakenberg J, Leser U: A comprehensive benchmark of kernel methods to extract protein-protein interactions from literature. PLoS Computational Biology. 2010, 6 (7): e1000837-10.1371\/journal.pcbi.1000837.","journal-title":"PLoS Computational Biology"},{"issue":"3","key":"654_CR23","doi-asserted-by":"publisher","first-page":"394","DOI":"10.1093\/bioinformatics\/btn631","volume":"25","author":"Y Miyao","year":"2009","unstructured":"Miyao Y, Sagae K, S\u00e6tre R, Matsuzaki T, Tsujii J: Evaluating contributions of natural language parsers to protein-protein interaction extraction. Bioinformatics. 2009, 25 (3): 394-400. 10.1093\/bioinformatics\/btn631.","journal-title":"Bioinformatics"},{"issue":"2002","key":"654_CR24","first-page":"3","volume":"15","author":"D Klein","year":"2002","unstructured":"Klein D, Manning C: Fast exact inference with a factored model for natural language parsing. Advances in neural information processing systems. 2002, 15 (2002): 3-10.","journal-title":"Advances in neural information processing systems"},{"issue":"Suppl 3","key":"654_CR25","doi-asserted-by":"publisher","first-page":"S6","DOI":"10.1186\/1471-2105-9-S3-S6","volume":"9","author":"S Pyysalo","year":"2008","unstructured":"Pyysalo S, Airola A, Heimonen J, Bj\u00f6rne J, Ginter F, Salakoski T: Comparative analysis of five protein-protein interaction corpora. BMC bioinformatics. 2008, 9 (Suppl 3): S6-10.1186\/1471-2105-9-S3-S6.","journal-title":"BMC bioinformatics"},{"key":"654_CR26","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1186\/1471-2105-8-50","volume":"8","author":"S Pyysalo","year":"2007","unstructured":"Pyysalo S, Ginter F, Heimonen J, Bjorne J, Boberg J, Jarvinen J, Salakoski T: BioInfer: a corpus for information extraction in the biomedical domain. BMC Bioinformatics. 2007, 8: 50-10.1186\/1471-2105-8-50.","journal-title":"BMC Bioinformatics"},{"key":"654_CR27","first-page":"326","volume-title":"Pacific Symposium on Biocomputing, Volume 7","author":"J Ding","year":"2002","unstructured":"Ding J, Berleant D, Nettleton D, Wurtele E: Mining MEDLINE: abstracts, sentences, or phrases. Pacific Symposium on Biocomputing, Volume 7. 2002, 326-337."},{"key":"654_CR28","first-page":"31","volume-title":"Proceedings of the 4th Learning Language in Logic Workshop (LLL05)","author":"C N\u00e9dellec","year":"2005","unstructured":"N\u00e9dellec C: Learning language in logic-genic interaction extraction challenge. Proceedings of the 4th Learning Language in Logic Workshop (LLL05). 2005, Citeseer, 31-37."},{"key":"654_CR29","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1038\/nmeth.1284","volume":"6","author":"M Cusick","year":"2009","unstructured":"Cusick M, Yu H, Smolyar A, Venkatesan K, Carvunis A, Simonis N, Rual J, Borick H, Braun P, Dreze M, et al: Literature-curated protein interaction datasets. Nature methods. 2009, 6: 39-46. 10.1038\/nmeth.1284.","journal-title":"Nature methods"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/1472-6947-13-S1-S7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/1472-6947-13-S1-S7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/1472-6947-13-S1-S7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T22:49:39Z","timestamp":1630536579000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/1472-6947-13-S1-S7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,4]]},"references-count":29,"journal-issue":{"issue":"S1","published-print":{"date-parts":[[2013,4]]}},"alternative-id":["654"],"URL":"https:\/\/doi.org\/10.1186\/1472-6947-13-s1-s7","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,4]]},"assertion":[{"value":"5 April 2013","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"S7"}}