{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T02:49:23Z","timestamp":1787885363286,"version":"build-2784847793"},"reference-count":58,"publisher":"Oxford University Press (OUP)","issue":"21","license":[{"start":{"date-parts":[[2019,4,2]],"date-time":"2019-04-02T00:00:00Z","timestamp":1554163200000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Database Integration Coordination Program"},{"DOI":"10.13039\/501100004696","name":"National Bioscience Database Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004696","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004696","name":"NBDC","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004696","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001695","name":"JST","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001695","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["LM009254"],"award-info":[{"award-number":["LM009254"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Most currently available text mining tools share two characteristics that make them less than optimal for use by biomedical researchers: they require extensive specialist skills in natural language processing and they were built on the assumption that they should optimize global performance metrics on representative datasets. This is a problem because most end-users are not natural language processing specialists and because biomedical researchers often care less about global metrics like F-measure or representative datasets than they do about more granular metrics such as precision and recall on their own specialized datasets. Thus, there are fundamental mismatches between the assumptions of much text mining work and the preferences of potential end-users.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>This article introduces the concept of Agile text mining, and presents the PubAnnotation ecosystem as an example implementation. The system approaches the problems from two perspectives: it allows the reformulation of text mining by biomedical researchers from the task of assembling a complete system to the task of retrieving warehoused annotations, and it makes it possible to do very targeted customization of the pre-existing system to address specific end-user requirements. Two use cases are presented: assisted curation of the GlycoEpitope database, and assessing coverage in the literature of pre-eclampsia-associated genes.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>The three tools that make up the ecosystem, PubAnnotation, PubDictionaries and TextAE are publicly available as web services, and also as open source projects. The dictionaries and the annotation datasets associated with the use cases are all publicly available through PubDictionaries and PubAnnotation, respectively.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz227","type":"journal-article","created":{"date-parts":[[2019,3,29]],"date-time":"2019-03-29T12:10:32Z","timestamp":1553861432000},"page":"4372-4380","source":"Crossref","is-referenced-by-count":28,"title":["Open Agile text mining for bioinformatics: the PubAnnotation ecosystem"],"prefix":"10.1093","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8877-3248","authenticated-orcid":false,"given":"Jin-Dong","family":"Kim","sequence":"first","affiliation":[{"name":"Database Center for Life Science, Research Organization of Information and Systems , Kashiwa, Chiba, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6230-3275","authenticated-orcid":false,"given":"Yue","family":"Wang","sequence":"additional","affiliation":[{"name":"Database Center for Life Science, Research Organization of Information and Systems , Kashiwa, Chiba, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0170-9172","authenticated-orcid":false,"given":"Toyofumi","family":"Fujiwara","sequence":"additional","affiliation":[{"name":"Database Center for Life Science, Research Organization of Information and Systems , Kashiwa, Chiba, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7704-8104","authenticated-orcid":false,"given":"Shujiro","family":"Okuda","sequence":"additional","affiliation":[{"name":"Graduate School of Medical and Dental Sciences, Niigata University , Niigata, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8169-9049","authenticated-orcid":false,"given":"Tiffany J","family":"Callahan","sequence":"additional","affiliation":[{"name":"Computational Bioscience Program, University of Colorado Denver, Anschutz Medical Campus , Aurora, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1749-8290","authenticated-orcid":false,"given":"K Bretonnel","family":"Cohen","sequence":"additional","affiliation":[{"name":"Computational Bioscience Program, University of Colorado Denver, Anschutz Medical Campus , Aurora, CO, USA"},{"name":"Universit\u00e9 Paris-Saclay, LIMSI-ILES , France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,4,1]]},"reference":[{"key":"2023062712462128600_btz227-B1","author":"Alex","year":"2008"},{"key":"2023062712462128600_btz227-B2","first-page":"429","article-title":"How much effort is needed to keep up with the literature relevant for primary care?","volume":"92","author":"Alper","year":"2004","journal-title":"J. Med. Libr. Assoc"},{"key":"2023062712462128600_btz227-B3","doi-asserted-by":"crossref","first-page":"S7.","DOI":"10.1186\/gb-2008-9-s2-s7","article-title":"Text mining for biology-the way forward: opinions from leading scientists","volume":"9","author":"Altman","year":"2008","journal-title":"Genome Biol"},{"key":"2023062712462128600_btz227-B4","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"2023062712462128600_btz227-B5","doi-asserted-by":"crossref","first-page":"214365","DOI":"10.1155\/2011\/214365","article-title":"Maternal preeclampsia and neonatal outcomes","volume":"2011","author":"Backes","year":"2011","journal-title":"J. Pregnancy"},{"key":"2023062712462128600_btz227-B6","doi-asserted-by":"crossref","first-page":"baw066","DOI":"10.1093\/database\/baw066","article-title":"Argo: enabling the development of bespoke workflows and services for disease annotation","volume":"2016","author":"Batista-Navarro","year":"2016","journal-title":"Database"},{"key":"2023062712462128600_btz227-B7","author":"Beck","year":"2001"},{"key":"2023062712462128600_btz227-B8","author":"Butte","year":"2006"},{"key":"2023062712462128600_btz227-B9","doi-asserted-by":"crossref","first-page":"27.","DOI":"10.1186\/s13020-017-0148-7","article-title":"Revealing topics and their evolution in biomedical literature using Bio-DTM: a case study of ginseng","volume":"12","author":"Chen","year":"2017","journal-title":"Chin. Med"},{"key":"2023062712462128600_btz227-B10","volume-title":"Foundations of Statistical Natural Language Processing","author":"Chris","year":"1999"},{"key":"2023062712462128600_btz227-B11","doi-asserted-by":"crossref","DOI":"10.1075\/nlp.11","volume-title":"Biomedical Natural Language Processing","author":"Cohen","year":"2014"},{"key":"2023062712462128600_btz227-B12","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1007\/978-1-60327-292-6_23","volume-title":"Mouse Molecular Embryology","author":"Condie","year":"2014"},{"key":"2023062712462128600_btz227-B13","doi-asserted-by":"crossref","first-page":"S120","DOI":"10.1016\/j.jbi.2015.06.030","article-title":"Agile text mining for the 2014 i2b2\/UTHealth cardiac risk factors challenge","volume":"58","author":"Cormack","year":"2015","journal-title":"J. Biomed. Inform"},{"key":"2023062712462128600_btz227-B14","doi-asserted-by":"crossref","first-page":"e17","DOI":"10.2196\/medinform.7123","article-title":"Validation of an improved computer-assisted technique for mining free-text electronic medical records","volume":"5","author":"Duz","year":"2017","journal-title":"JMIR Med. Inform"},{"key":"2023062712462128600_btz227-B15","first-page":"241","volume-title":"Informatics for Health: Connected Citizen-Led Wellness and Population Health","author":"Dziadek","year":"2017"},{"key":"2023062712462128600_btz227-B16","doi-asserted-by":"crossref","first-page":"59.","DOI":"10.1186\/1471-2105-15-59","article-title":"Large-scale biomedical concept recognition: an evaluation of current automatic annotators and their parameters","volume":"15","author":"Funk","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2023062712462128600_btz227-B17","doi-asserted-by":"crossref","first-page":"344","DOI":"10.3758\/s13428-017-0875-9","article-title":"Dictionaries and distributions: combining expert knowledge and large scale textual data content analysis","volume":"50","author":"Garten","year":"2018","journal-title":"Behav. Res. Methods"},{"key":"2023062712462128600_btz227-B18","doi-asserted-by":"crossref","first-page":"S6","DOI":"10.1186\/1471-2105-10-S2-S6","article-title":"Pharmspresso: a text mining tool for extraction of pharmacogenomic concepts and relationships from full text","volume":"10","author":"Garten","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2023062712462128600_btz227-B19","doi-asserted-by":"crossref","DOI":"10.1542\/peds.2016-0985","article-title":"Development of a pediatric adverse events terminology","volume":"139","author":"Gipson","year":"2017","journal-title":"Pediatrics"},{"key":"2023062712462128600_btz227-B20","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1146\/annurev.iy.02.040184.000535","article-title":"Tumor-associated carbohydrate antigens","volume":"2","author":"Hakomori","year":"1984","journal-title":"Annu. Rev. Immunol"},{"key":"2023062712462128600_btz227-B21","doi-asserted-by":"crossref","first-page":"bas020","DOI":"10.1093\/database\/bas020","article-title":"Text mining for the biocuration workflow","volume":"2012","author":"Hirschman","year":"2012","journal-title":"Database"},{"key":"2023062712462128600_btz227-B22","article-title":"Tagger: BeCalm API for rapid named entity recognition","author":"Jensen","year":"2017","journal-title":"bioRxiv"},{"key":"2023062712462128600_btz227-B23","doi-asserted-by":"crossref","first-page":"44.","DOI":"10.1186\/s13326-017-0153-x","article-title":"Semantic annotation in biomedicine: the current landscape","volume":"8","author":"Jovanovi","year":"2017","journal-title":"J. Biomed. Semantics"},{"key":"2023062712462128600_btz227-B24","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/j.jbi.2016.01.008","article-title":"Toward better public health reporting using existing off the shelf approaches: a comparison of alternative cancer detection approaches using plaintext medical data and non-dictionary based feature selection","volume":"60","author":"Kasthurirathne","year":"2016","journal-title":"J. Biomed. Inform"},{"key":"2023062712462128600_btz227-B25","doi-asserted-by":"crossref","first-page":"267","DOI":"10.4052\/tigg.18.267","article-title":"GlycoEpitope: the integrated database of carbohydrate antigens and antibodies","volume":"18","author":"Kawasaki","year":"2006","journal-title":"Trends Glycosci. Glycotechnol"},{"key":"2023062712462128600_btz227-B26","volume-title":"Selected abstracts from the 1st Biomedical Linked Annotation Hackathon (BLAH1)","author":"Kim","year":"2015"},{"key":"2023062712462128600_btz227-B27","first-page":"202","volume-title":"Proceedings of the 2012 Workshop on Biomedical Natural Language Processing","author":"Kim","year":"2012"},{"key":"2023062712462128600_btz227-B28","doi-asserted-by":"crossref","first-page":"331.","DOI":"10.1186\/1756-0500-4-331","article-title":"Agile parallel bioinformatics workflow management using Pwrake","volume":"4","author":"Mishima","year":"2011","journal-title":"BMC Res. Notes"},{"key":"2023062712462128600_btz227-B29","doi-asserted-by":"crossref","first-page":"e0162828.","DOI":"10.1371\/journal.pone.0162828","article-title":"The pre-eclampsia ontology: a disease ontology representing the domain knowledge specific to pre-eclampsia","volume":"11","author":"Mizuno","year":"2016","journal-title":"PLoS One"},{"key":"2023062712462128600_btz227-B30","doi-asserted-by":"crossref","first-page":"e309.","DOI":"10.1371\/journal.pbio.0020309","article-title":"Textpresso: an ontology-based information retrieval and extraction system for biological literature","volume":"2","author":"M\u00fcller","year":"2004","journal-title":"PLoS Biol"},{"key":"2023062712462128600_btz227-B31","doi-asserted-by":"crossref","first-page":"94.","DOI":"10.1186\/s12859-018-2103-8","article-title":"Textpresso central: a customizable platform for searching, text mining, viewing, and curating biomedical literature","volume":"19","author":"M\u00fcller","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2023062712462128600_btz227-B32","first-page":"24","article-title":"TheQuaero French medical corpus: A resource for medical entity recognition and normalization","volume-title":"Proceedings of Fourth Workshop on Building and EvaluatingResources for Health and Biomedical Text Processing","author":"N\u00e9v\u00e9ol","year":"2014"},{"key":"2023062712462128600_btz227-B33","doi-asserted-by":"crossref","first-page":"e29626","DOI":"10.3897\/BDJ.7.e29626","article-title":"Copious: a gold standard corpus of named entities towards extracting species occurrence from biodiversity literature","volume":"7","author":"Nguyen","year":"2019","journal-title":"Biodivers. Data J"},{"key":"2023062712462128600_btz227-B34","doi-asserted-by":"crossref","first-page":"e88889.","DOI":"10.1371\/journal.pone.0088889","article-title":"Functionally enigmatic genes: a case study of the brain ignorome","volume":"9","author":"Pandey","year":"2014","journal-title":"PLoS One"},{"key":"2023062712462128600_btz227-B35","first-page":"149","author":"Papazian","year":"2012"},{"key":"2023062712462128600_btz227-B36","doi-asserted-by":"crossref","first-page":"151.","DOI":"10.1186\/s12872-017-0580-8","article-title":"Unlocking echocardiogram measurements for heart disease research through natural language processing","volume":"17","author":"Patterson","year":"2017","journal-title":"BMC Cardiovasc. Disord"},{"key":"2023062712462128600_btz227-B37","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1007\/s40264-016-0491-0","article-title":"Evaluation of Facebook and twitter monitoring to detect safety signals for medical products: an analysis of recent FDA safety alerts","volume":"40","author":"Pierce","year":"2017","journal-title":"Drug Saf"},{"key":"2023062712462128600_btz227-B38","doi-asserted-by":"crossref","first-page":"bas010","DOI":"10.1093\/database\/bas010","article-title":"Argo: an integrative, interactive, text mining-based workbench supporting curation","volume":"2012","author":"Rak","year":"2012","journal-title":"Database"},{"key":"2023062712462128600_btz227-B39","doi-asserted-by":"crossref","first-page":"bau070","DOI":"10.1093\/database\/bau070","article-title":"Text-mining-assisted biocuration workflows in Argo","volume":"2014","author":"Rak","year":"2014","journal-title":"Database"},{"key":"2023062712462128600_btz227-B40","doi-asserted-by":"crossref","first-page":"24647","DOI":"10.1038\/srep24647","article-title":"Revealing the acute asthma ignorome: characterization and validation of uninvestigated gene networks","volume":"6","author":"Riba","year":"2016","journal-title":"Sci. Rep"},{"key":"2023062712462128600_btz227-B41","doi-asserted-by":"crossref","first-page":"1800","DOI":"10.1093\/bioinformatics\/btq250","article-title":"A UIMA wrapper for the NCBO annotator","volume":"26","author":"Roeder","year":"2010","journal-title":"Bioinformatics"},{"key":"2023062712462128600_btz227-B42","first-page":"75","author":"Shivade","year":"2014"},{"key":"2023062712462128600_btz227-B43","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1016\/S0140-6736(05)17987-2","article-title":"Pre-eclampsia","volume":"365","author":"Sibai","year":"2005","journal-title":"Lancet"},{"key":"2023062712462128600_btz227-B44","author":"Silberztein","year":"2005"},{"key":"2023062712462128600_btz227-B45","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1093\/bioinformatics\/btv584","article-title":"SUPER-FOCUS: a tool for agile functional analysis of shotgun metagenomic data","volume":"32","author":"Silva","year":"2016","journal-title":"Bioinformatics"},{"key":"2023062712462128600_btz227-B46","first-page":"102","author":"Stenetorp","year":"2012"},{"key":"2023062712462128600_btz227-B47","author":"Stubbs","year":"2013"},{"key":"2023062712462128600_btz227-B48","doi-asserted-by":"crossref","first-page":"3138","DOI":"10.1093\/bioinformatics\/btq597","article-title":"GSearcher: agile attribute querying for biological networks","volume":"26","author":"Su","year":"2010","journal-title":"Bioinformatics"},{"key":"2023062712462128600_btz227-B49","first-page":"42","author":"Sukkarieh","year":"2009"},{"key":"2023062712462128600_btz227-B50","doi-asserted-by":"crossref","first-page":"1124","DOI":"10.1093\/bioinformatics\/18.8.1124","article-title":"Tagging gene and protein names in biomedical text","volume":"18","author":"Tanabe","year":"2002","journal-title":"Bioinformatics"},{"key":"2023062712462128600_btz227-B51","first-page":"57","author":"Thomas","year":"2007"},{"key":"2023062712462128600_btz227-B52","doi-asserted-by":"crossref","first-page":"bas040","DOI":"10.1093\/database\/bas040","article-title":"Text mining in the biocuration workflow: applications for literature curation at WormBase, dictyBase and TAIR","volume":"2012","author":"Van Auken","year":"2012","journal-title":"Database"},{"key":"2023062712462128600_btz227-B53","doi-asserted-by":"crossref","first-page":"baw119","DOI":"10.1093\/database\/baw119","article-title":"Overview of the interactive task in BioCreative V","volume":"2016","author":"Wang","year":"2016","journal-title":"Database"},{"key":"2023062712462128600_btz227-B54","first-page":"1","author":"Wang","year":"2018"},{"key":"2023062712462128600_btz227-B55","doi-asserted-by":"crossref","first-page":"W518","DOI":"10.1093\/nar\/gkt441","article-title":"Pubtator: a web-based text mining tool for assisting biocuration","volume":"41","author":"Wei","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2023062712462128600_btz227-B56","doi-asserted-by":"crossref","first-page":"S2.","DOI":"10.1186\/1471-2105-6-S1-S2","article-title":"Biocreative task 1a: gene mention finding evaluation","volume":"6","author":"Yeh","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2023062712462128600_btz227-B57","doi-asserted-by":"crossref","first-page":"1997","DOI":"10.1093\/bioinformatics\/btp289","article-title":"U-Compare: share and compare text mining tools with UIMA","volume":"25","author":"Yoshinobu","year":"2009","journal-title":"Bioinformatics"},{"key":"2023062712462128600_btz227-B58","doi-asserted-by":"crossref","first-page":"481.","DOI":"10.1186\/1471-2105-12-481","article-title":"U-Compare bio-event meta-service: compatible BioNLP event extraction services","volume":"12","author":"Yoshinobu","year":"2011","journal-title":"BMC Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btz227\/28554795\/btz227.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/21\/4372\/50722037\/bioinformatics_35_21_4372.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/21\/4372\/50722037\/bioinformatics_35_21_4372.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,27]],"date-time":"2023-06-27T12:47:23Z","timestamp":1687870043000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/35\/21\/4372\/5425327"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2019,4,1]]},"references-count":58,"journal-issue":{"issue":"21","published-print":{"date-parts":[[2019,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz227","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,11,1]]},"published":{"date-parts":[[2019,4,1]]}}}