{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T10:01:37Z","timestamp":1771063297788,"version":"3.50.1"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2007,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Knowledge base construction has been an area of intense activity and great importance in the growth of computational biology. However, there is little or no history of work on the subject of evaluation of knowledge bases, either with respect to their contents or with respect to the processes by which they are constructed. This article proposes the application of a metric from software engineering known as the found\/fixed graph to the problem of evaluating the processes by which genomic knowledge bases are built, as well as the completeness of their contents.<\/jats:p>\n               <jats:p>Results: Well-understood patterns of change in the found\/fixed graph are found to occur in two large publicly available knowledge bases. These patterns suggest that the current manual curation processes will take far too long to complete the annotations of even just the most important model organisms, and that at their current rate of production, they will never be sufficient for completing the annotation of all currently available proteomes.<\/jats:p>\n               <jats:p>Contact: larry.hunter@uchsc.edu<\/jats:p>","DOI":"10.1093\/bioinformatics\/btm229","type":"journal-article","created":{"date-parts":[[2007,7,23]],"date-time":"2007-07-23T16:13:46Z","timestamp":1185207226000},"page":"i41-i48","source":"Crossref","is-referenced-by-count":184,"title":["Manual curation is not sufficient for annotation of genomic databases"],"prefix":"10.1093","volume":"23","author":[{"suffix":"Jr","given":"William A.","family":"Baumgartner","sequence":"first","affiliation":[{"name":"1 Center for Computational Pharmacology, University of Colorado School of Medicine, 2Denison Library, University of Colorado Health Science Center and 3Department of Pharmaceutical Sciences, Massachusetts College of Pharmacy and Health Sciences, USA"}]},{"given":"K. Bretonnel","family":"Cohen","sequence":"additional","affiliation":[{"name":"1 Center for Computational Pharmacology, University of Colorado School of Medicine, 2Denison Library, University of Colorado Health Science Center and 3Department of Pharmaceutical Sciences, Massachusetts College of Pharmacy and Health Sciences, USA"}]},{"given":"Lynne M.","family":"Fox","sequence":"additional","affiliation":[{"name":"1 Center for Computational Pharmacology, University of Colorado School of Medicine, 2Denison Library, University of Colorado Health Science Center and 3Department of Pharmaceutical Sciences, Massachusetts College of Pharmacy and Health Sciences, USA"}]},{"given":"George","family":"Acquaah-Mensah","sequence":"additional","affiliation":[{"name":"1 Center for Computational Pharmacology, University of Colorado School of Medicine, 2Denison Library, University of Colorado Health Science Center and 3Department of Pharmaceutical Sciences, Massachusetts College of Pharmacy and Health Sciences, USA"}]},{"given":"Lawrence","family":"Hunter","sequence":"additional","affiliation":[{"name":"1 Center for Computational Pharmacology, University of Colorado School of Medicine, 2Denison Library, University of Colorado Health Science Center and 3Department of Pharmaceutical Sciences, Massachusetts College of Pharmacy and Health Sciences, USA"}]}],"member":"286","published-online":{"date-parts":[[2007,7,1]]},"reference":[{"key":"2023062708514941200_B1","article-title":"Design and implementation of a knowledge-base for pharmacology","volume-title":"In Proceedings of the 5th Annual Bio-Ontologies Meeting","author":"Acquaah-Mensah","year":"2002"},{"issue":"Database issue","key":"2023062708514941200_B2","doi-asserted-by":"crossref","first-page":"D322","DOI":"10.1093\/nar\/gkl799","article-title":"GO PaD: the Gene Ontology Partition Database","volume":"35","author":"Alterovitz","year":"2007","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B3","first-page":"309","article-title":"Collaborative curation of data from bio-medical texts and abstracts and its integration","author":"Baral","year":"2005"},{"key":"2023062708514941200_B4","volume-title":"Software Testing Techniques","author":"Beizer","year":"1990","edition":"2nd"},{"key":"2023062708514941200_B5","volume-title":"Black-Box Testing: Techniques for Functional Testing of Software and Systems","author":"Beizer","year":"1995"},{"key":"2023062708514941200_B6","volume-title":"Managing the Software Testing Process","author":"Black","year":"1999"},{"key":"2023062708514941200_B7","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1093\/nar\/gkg095","article-title":"The SWISS-PROT protein knowledgebase and its supplement TrEMBL in 2003","volume":"31","author":"Boeckmann","year":"2003","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B8","doi-asserted-by":"crossref","first-page":"933","DOI":"10.1038\/35023188","article-title":"Sequencing solution: use volunteer annotators organized via Internet","volume":"406","author":"Brinkman","year":"2000","journal-title":"Nature"},{"key":"2023062708514941200_B9","doi-asserted-by":"crossref","first-page":"e99","DOI":"10.1371\/journal.pcbi.0020099","article-title":"A biocurator perspective: annotation at the Research Collaboratory for Structural Bioinformatics Protein Data Bank","volume":"2","author":"Burkhardt","year":"2006","journal-title":"PLoS Comput Biol"},{"key":"2023062708514941200_B10","doi-asserted-by":"crossref","first-page":"D262","DOI":"10.1093\/nar\/gkh021","article-title":"The Gene Ontology Annotation (GOA) Database: sharing knowledge in UniProt with Gene Ontology","volume":"32","author":"Camon","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B11","article-title":"Mistakes in medical ontologies: where do they come from and how can they be detected?","volume-title":"Ontologies in Medicine: Proceedings of the Workshop on Medical Ontologies","author":"Ceusters","year":"2003"},{"key":"2023062708514941200_B12","first-page":"84","article-title":"RIBOWEB: linking structural computations to a knowledge base of published experimental data","volume-title":"Proc. Intell. Syst. Mol. Biol","author":"Chen","year":"1999"},{"key":"2023062708514941200_B13","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1016\/j.jbi.2003.11.001","article-title":"Consistency across the hierarchies of the UMLS Semantic Network and Metathesaurus","volume":"36","author":"Cimino","year":"2003","journal-title":"J. Biomed. Informatics"},{"key":"2023062708514941200_B14","volume-title":"Empirical methods for artificial intelligence","author":"Cohen","year":"1995"},{"key":"2023062708514941200_B15","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1038\/445229b","article-title":"The database revolution","volume":"445","author":"Editorial","year":"2007","journal-title":"Nature"},{"key":"2023062708514941200_B16","doi-asserted-by":"crossref","first-page":"1425","DOI":"10.1101\/gr.180801","article-title":"Creating the Gene Ontology resource: design and implementation","volume":"11","author":"Gene Ontology Consortium","year":"2001","journal-title":"Genome Res"},{"key":"2023062708514941200_B17","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1038\/445691a","article-title":"Key biology databases go wiki","volume":"445","author":"Giles","year":"2007","journal-title":"Nature"},{"key":"2023062708514941200_B18","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1136\/jamia.1995.96073832","article-title":"Evaluation of long-term maintenance of a large medical knowledge base","volume":"2","author":"Giuse","year":"1995","journal-title":"J. Am. Med. Assoc"},{"key":"2023062708514941200_B19","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1093\/nar\/gkg125","article-title":"ASAP, a systematic annotation package for community analysis of genomes","volume":"31","author":"Glasner","year":"2003","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B20","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1007\/s10115-003-0140-7","article-title":"A quantitative analysis of the robustness of knowledge-based systems through degradation studies","volume":"7","author":"Groot","year":"2005","journal-title":"Knowledge Information Syst"},{"key":"2023062708514941200_B21","first-page":"14","article-title":"TREC Genomics track overview","volume-title":"Proc. TREC 2003","author":"Hersh","year":"2003"},{"key":"2023062708514941200_B22","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1093\/nar\/30.1.163","article-title":"PharmGKB: the Pharmacogenetics Knowledge Base","volume":"30","author":"Hewett","year":"2002","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B23","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1093\/bioinformatics\/btg449","article-title":"Automated extraction of mutation data from the literature: application of MuteXt to G protein-coupled receptors and nuclear hormone receptors","volume":"20","author":"Horn","year":"2004","journal-title":"Bioinformatics"},{"key":"2023062708514941200_B24","volume-title":"Testing computer software","author":"Kaner","year":"1999","edition":"2nd"},{"key":"2023062708514941200_B25","volume-title":"Lessons learned in software testing","author":"Kaner","year":"2001"},{"key":"2023062708514941200_B26","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-7-212","article-title":"Quality control for terms and definitions in ontologies and taxonomies","volume":"7","author":"K\u00f6hler","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023062708514941200_B27","first-page":"601","article-title":"Semantic similarity measures as tools for exploring the Gene Ontology","volume":"8","author":"Lord","year":"2003","journal-title":"Pacific Symp. Biocomput"},{"key":"2023062708514941200_B28","doi-asserted-by":"crossref","first-page":"1275","DOI":"10.1093\/bioinformatics\/btg153","article-title":"Investigating semantic similarity measures across the Gene Ontology: the relationship between sequence and annotation","volume":"19","author":"Lord","year":"2003","journal-title":"Bioinformatics"},{"key":"2023062708514941200_B29","first-page":"52","article-title":"Finding GeneRIFs via Gene Ontology annotations","volume":"11","author":"Lu","year":"2006","journal-title":"Pac. Symp. Biocomput"},{"key":"2023062708514941200_B30","first-page":"269","article-title":"GeneRIF quality assurance as summary revision","volume":"12","author":"Lu","year":"2007","journal-title":"Pac. Symp. on Biocomput"},{"issue":"Database Issue","key":"2023062708514941200_B31","doi-asserted-by":"crossref","first-page":"D54","DOI":"10.1093\/nar\/gki031","article-title":"Entrez Gene: gene-centered information at NCBI","volume":"33","author":"Maglott","year":"2005","journal-title":"Nucleic Acids Res"},{"key":"2023062708514941200_B32","first-page":"460","article-title":"Gene indexing: characterization and analysis of NLM's GeneRIFs","volume-title":"AMIA Annual Symposium Proc","author":"Mitchell","year":"2003"},{"key":"2023062708514941200_B33","volume-title":"The Art of Software Testing","author":"Myers","year":"1979"},{"key":"2023062708514941200_B34","doi-asserted-by":"crossref","DOI":"10.1007\/978-94-009-2213-6","volume-title":"Mathematical methods in linguistics","author":"Partee","year":"1993","edition":"corrected 1st"},{"key":"2023062708514941200_B35","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-6-12","article-title":"MILANO\u2014custom annotation of microarray results using automatic literature searches","volume":"6","author":"Rubinstein","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2023062708514941200_B36","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1186\/gb-2007-8-1-102","article-title":"Opinion: Genome re-annotation: a wiki solution?","volume":"8","author":"Salzberg","year":"2007","journal-title":"Genome Biol"},{"key":"2023062708514941200_B37","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.tplants.2004.11.002","article-title":"Community-based gene structure annotation","volume":"10","author":"Schlueter","year":"2005","journal-title":"Trends Plant Sci"},{"key":"2023062708514941200_B38","doi-asserted-by":"crossref","first-page":"R111","DOI":"10.1186\/gb-2006-7-11-r111","article-title":"xGDB: open-source computational infrastructure for the integrated evaluation and analysis of genome features","volume":"7","author":"Schlueter","year":"2006","journal-title":"Genome Biol"},{"key":"2023062708514941200_B39","first-page":"345","article-title":"Building large knowledge bases in molecular biology","volume-title":"Proc. Intel. Sys. Mol. Biol","author":"Schmeltzer","year":"1993"},{"key":"2023062708514941200_B40","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1186\/1471-2105-8-17","article-title":"Publishing perishing? Towards tomorrow's information architecture","volume":"8","author":"Seringhaus","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023062708514941200_B41","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1371\/journal.pcbi.0010010","article-title":"Extraction of transcript diversity from scientific literature","volume":"1","author":"Shah","year":"2005","journal-title":"PLoS Computational Biology"},{"key":"2023062708514941200_B42","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1038\/35023079","article-title":"Complete genome sequence of Pseudomonas aeruginosa PA01, an opportunistic pathogen","volume":"406","author":"Stover","year":"2000","journal-title":"Nature"},{"key":"2023062708514941200_B43","first-page":"900","article-title":"Comment: Gene-function wiki would let biologists pool worldwide resources","volume":"438","author":"Wang","year":"2006","journal-title":"Nature"},{"key":"2023062708514941200_B44","doi-asserted-by":"crossref","first-page":"R58","DOI":"10.1186\/gb-2006-7-7-r58","article-title":"yrGATE: a web-based gene-structure annotation tool for the identification and dissemination of eukaryotic genes","volume":"7","author":"Wilkerson","year":"2006","journal-title":"Genome Biol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/23\/13\/i41\/50718269\/bioinformatics_23_13_i41.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/23\/13\/i41\/50718269\/bioinformatics_23_13_i41.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,27]],"date-time":"2023-06-27T08:57:30Z","timestamp":1687856250000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/23\/13\/i41\/238103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,7,1]]},"references-count":44,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2007,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btm229","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2007,7]]},"published":{"date-parts":[[2007,7,1]]}}}