{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:11:03Z","timestamp":1772165463719,"version":"3.50.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,25]],"date-time":"2021-11-25T00:00:00Z","timestamp":1637798400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,25]],"date-time":"2021-11-25T00:00:00Z","timestamp":1637798400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP190101350"],"award-info":[{"award-number":["DP190101350"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Literature-based gene ontology (GO) annotation is a process where expert curators use uniform expressions to describe gene functions reported in research papers, creating computable representations of information about biological systems. Manual assurance of consistency between GO annotations and the associated evidence texts identified by expert curators is reliable but time-consuming, and is infeasible in the context of rapidly growing biological literature. A key challenge is maintaining consistency of existing GO annotations as new studies are published and the GO vocabulary is updated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In this work, we introduce a formalisation of biological database annotation inconsistencies, identifying four distinct types of inconsistency. We propose a novel and efficient method using state-of-the-art text mining models to automatically distinguish between consistent GO annotation and the different types of inconsistent GO annotation. We evaluate this method using a synthetic dataset generated by directed manipulation of instances in an existing corpus, BC4GO. We provide detailed error analysis for demonstrating that the method achieves high precision on more confident predictions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Two models built using our method for distinct annotation consistency identification tasks achieved high precision and were robust to updates in the GO vocabulary. Our approach demonstrates clear value for human-in-the-loop curation scenarios.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12859-021-04479-9","type":"journal-article","created":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T13:32:03Z","timestamp":1638279123000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Automatic consistency assurance for literature-based gene ontology annotation"],"prefix":"10.1186","volume":"22","author":[{"given":"Jiyu","family":"Chen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicholas","family":"Geard","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin","family":"Zobel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8661-1544","authenticated-orcid":false,"given":"Karin","family":"Verspoor","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,25]]},"reference":[{"issue":"D1","key":"4479_CR1","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1093\/nar\/gku1179","volume":"43","author":"Gene Ontology Consortium","year":"2015","unstructured":"Gene Ontology Consortium. Gene ontology consortium: going forward. Nucleic Acids Res. 2015;43(D1):1049\u201356.","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"4479_CR2","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1038\/75556","volume":"25","author":"M Ashburner","year":"2000","unstructured":"Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, et al. Gene ontology: tool for the unification of biology. Nat Genet. 2000;25(1):25\u20139.","journal-title":"Nat Genet"},{"key":"4479_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13059-018-1612-0","volume":"20","author":"N Zhou","year":"2019","unstructured":"Zhou N, Jiang Y, Bergquist TR. The CAFA challenge reports improved protein function prediction and new functional. Genome Biol. 2019;20:1.","journal-title":"Genome Biol"},{"key":"4479_CR4","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1007\/978-1-4939-3743-1_5","volume":"1446","author":"D Cozzetto","year":"2017","unstructured":"Cozzetto D, Jones D. Computational methods for annotation transfers from sequence. Methods Mol Biol (Clifton, NJ). 2017;1446:55\u201367.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"issue":"D1","key":"4479_CR5","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/nar\/gkw1108","volume":"45","author":"Gene Ontology Consortium","year":"2017","unstructured":"Gene Ontology Consortium. Expansion of the gene ontology knowledgebase and resources. Nucleic Acids Res. 2017;45(D1):331\u20138.","journal-title":"Nucleic Acids Res"},{"key":"4479_CR6","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1007\/978-1-4939-3743-1_6","volume":"1446","author":"P Ruch","year":"2017","unstructured":"Ruch P. Text mining to support gene ontology curation and vice versa. Methods Mol Biol (Clifton, NJ). 2017;1446:69\u201384.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"key":"4479_CR7","doi-asserted-by":"crossref","unstructured":"Balakrishnan R, Harris MA, Huntley R, Van Auken K, Cherry JM. A guide to best practices for gene ontology (go) manual annotation. Database. 2013.","DOI":"10.1093\/database\/bat054"},{"issue":"6","key":"4479_CR8","doi-asserted-by":"publisher","first-page":"723","DOI":"10.1093\/bib\/bbr002","volume":"12","author":"L Du Plessis","year":"2011","unstructured":"Du Plessis L, \u0160kunca N, Dessimoz C. The what, where, how and why of gene ontology-a primer for bioinformaticians. Brief Bioinform. 2011;12(6):723\u201335.","journal-title":"Brief Bioinform"},{"issue":"5","key":"4479_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1002533","volume":"8","author":"N \u0160kunca","year":"2012","unstructured":"\u0160kunca N, Altenhoff A, Dessimoz C, et al. Quality of computationally inferred gene ontology annotations. PLOS Comput Biol. 2012;8(5):1\u201311.","journal-title":"PLOS Comput Biol"},{"key":"4479_CR10","doi-asserted-by":"crossref","unstructured":"Gobeill J, Pasche E, Vishnyakova D, Ruch P. Managing the data deluge: data-driven go category assignment improves while complexity of functional annotation increases. Database. 2013.","DOI":"10.1093\/database\/bat041"},{"key":"4479_CR11","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1007\/978-1-4939-3743-1_14","volume":"1446","author":"P Gaudet","year":"2017","unstructured":"Gaudet P, Dessimoz C. Gene ontology: pitfalls, biases, and remedies. Methods Mol Biol (Clifton, NJ). 2017;1446:189.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"issue":"1","key":"4479_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-018-19333-x","volume":"8","author":"WA Haynes","year":"2018","unstructured":"Haynes WA, Tomczak A, Khatri P. Gene annotation bias impedes biomedical research. Sci Rep. 2018;8(1):1\u20137.","journal-title":"Sci Rep"},{"issue":"5","key":"4479_CR13","doi-asserted-by":"publisher","first-page":"e1003063","DOI":"10.1371\/journal.pcbi.1003063","volume":"9","author":"AM Schnoes","year":"2013","unstructured":"Schnoes AM, Ream DC, Thorman AW, Babbitt PC, Friedberg I. Biases in the experimental annotations of protein function and their effect on our understanding of protein function space. PLoS Comput Biol. 2013;9(5):e1003063.","journal-title":"PLoS Comput Biol"},{"key":"4479_CR14","unstructured":"Tanenblatt M, Coden A, Sominsky I. The conceptmapper approach to named entity recognition. In: Proceedings of the seventh international conference on language resources and evaluation (LREC\u201910). 2010."},{"issue":"1","key":"4479_CR15","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1186\/1471-2105-13-161","volume":"13","author":"M Bada","year":"2012","unstructured":"Bada M, Eckert M, Evans D, Garcia K, Shipley K, Sitnikov D, Baumgartner WA, Cohen KB, Verspoor K, Blake JA, et al. Concept annotation in the craft corpus. BMC Bioinform. 2012;13(1):161.","journal-title":"BMC Bioinform"},{"issue":"1","key":"4479_CR16","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1186\/1471-2105-15-59","volume":"15","author":"C Funk","year":"2014","unstructured":"Funk C, Baumgartner W, Garcia B, Roeder C, Bada M, Cohen KB, Hunter LE, Verspoor K. Large-scale biomedical concept recognition: an evaluation of current automatic annotators and their parameters. BMC Bioinform. 2014;15(1):59.","journal-title":"BMC Bioinform"},{"issue":"12","key":"4479_CR17","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.1093\/bioinformatics\/18.12.1641","volume":"18","author":"WR Gilks","year":"2002","unstructured":"Gilks WR, Audit B, De Angelis D, Tsoka S, Ouzounis CA. Modeling the percolation of annotation errors in a database of protein sequences. Bioinformatics. 2002;18(12):1641\u20139.","journal-title":"Bioinformatics"},{"key":"4479_CR18","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1007\/978-1-4939-3743-1_8","volume":"1446","author":"N \u0160kunca","year":"2017","unstructured":"\u0160kunca N, Roberts R, Steffen M. Evaluating computational gene ontology annotations. Methods Mol Biol (Clifton, NJ). 2017;1446:97.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"issue":"1","key":"4479_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-8-170","volume":"8","author":"CE Jones","year":"2007","unstructured":"Jones CE, Brown AL, Baumann U. Estimating the annotation error rate of curated go database sequence annotations. BMC Bioinform. 2007;8(1):1\u20139.","journal-title":"BMC Bioinform"},{"key":"4479_CR20","doi-asserted-by":"crossref","unstructured":"Mao Y, Van Auken K, Li D, Arighi CN, McQuilton P, Hayman GT, Tweedie S, Schaeffer ML, Laulederkind SJ, Wang S-J, et al. Overview of the gene ontology task at biocreative IV. Database. 2014.","DOI":"10.1093\/database\/bau086"},{"key":"4479_CR21","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1007\/978-1-4939-3743-1_4","volume":"1446","author":"S Poux","year":"2017","unstructured":"Poux S, Gaudet P. Best practices in manual annotation with the gene ontology. Methods Mol Biol (Clifton, NJ). 2017;1446:41\u201354.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"key":"4479_CR22","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1007\/978-1-4939-3743-1_2","volume":"1446","author":"P Thomas","year":"2017","unstructured":"Thomas P. The gene ontology and the meaning of biological function. Methods Mol Biol (Clifton, NJ). 2017;1446:15\u201324.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"key":"4479_CR23","doi-asserted-by":"crossref","unstructured":"Van Auken K, Schaeffer ML, McQuilton P, Laulederkind SJ, Li D, Wang S-J, Hayman GT, Tweedie S, Arighi CN, Done J, et al. Bc4go: a full-text corpus for the biocreative IV go task. Database. 2014.","DOI":"10.1093\/database\/bau074"},{"issue":"1","key":"4479_CR24","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, Naumann T, Gao J, Poon H. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc (HEALTH). 2021;3(1):1\u201323.","journal-title":"ACM Trans Comput Healthc (HEALTH)"},{"key":"4479_CR25","unstructured":"Cohen KB, Roeder C, Baumgartner Jr WA, Hunter L, Verspoor K. Test suite design for biomedical ontology concept recognition systems. In: Proceedings of the seventh international conference on language resources and evaluation (LREC\u201910). 2010."},{"key":"4479_CR26","unstructured":"Groza T, Verspoor K. Automated generation of test suites for error analysis of concept recognition systems. In: Proceedings of the Australasian language technology association workshop. 2014. pp. 23\u201331."},{"key":"4479_CR27","doi-asserted-by":"crossref","unstructured":"Chen Q, Peng Y, Lu Z. Biosentvec: creating sentence embeddings for biomedical texts. In: 2019 IEEE international conference on healthcare informatics (ICHI). IEEE; 2019. p. 1\u20135.","DOI":"10.1109\/ICHI.2019.8904728"},{"issue":"1","key":"4479_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson AE, Pollard TJ, Shen L, Li-Wei HL, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3(1):1\u20139.","journal-title":"Sci Data"},{"issue":"14","key":"4479_CR29","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1093\/bioinformatics\/btx238","volume":"33","author":"G So\u011fanc\u0131o\u011flu","year":"2017","unstructured":"So\u011fanc\u0131o\u011flu G, \u00d6zt\u00fcrk H, \u00d6zg\u00fcr A. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics. 2017;33(14):49\u201358.","journal-title":"Bioinformatics"},{"issue":"1","key":"4479_CR30","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1007\/s10579-018-9431-1","volume":"54","author":"Y Wang","year":"2020","unstructured":"Wang Y, Afzal N, Fu S, Wang L, Shen F, Rastegar-Mojarad M, Liu H. Medsts: a resource for clinical semantic textual similarity. Lang Resour Eval. 2020;54(1):57\u201372.","journal-title":"Lang Resour Eval"},{"issue":"5","key":"4479_CR31","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1016\/j.jbi.2013.07.011","volume":"46","author":"M Herrero-Zazo","year":"2013","unstructured":"Herrero-Zazo M, Segura-Bedmar I, Mart\u00ednez P, Declerck T. The DDI corpus: an annotated corpus with pharmacological substances and drug\u2013drug interactions. J Biomed Inform. 2013;46(5):914\u201320.","journal-title":"J Biomed Inform"},{"issue":"5","key":"4479_CR32","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1038\/ng0504-431","volume":"36","author":"KG Becker","year":"2004","unstructured":"Becker KG, Barnes KC, Bright TJ, Wang SA. The genetic association database. Nat Genet. 2004;36(5):431\u20132.","journal-title":"Nat Genet"},{"key":"4479_CR33","unstructured":"Settles B. Active learning literature survey. 2009."},{"issue":"1","key":"4479_CR34","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1145\/584091.584093","volume":"5","author":"CE Shannon","year":"2001","unstructured":"Shannon CE. A mathematical theory of communication. ACM SIGMOBILE Mob Comput Commun Rev. 2001;5(1):3\u201355.","journal-title":"ACM SIGMOBILE Mob Comput Commun Rev"},{"issue":"22","key":"4479_CR35","doi-asserted-by":"publisher","first-page":"3045","DOI":"10.1093\/bioinformatics\/btp536","volume":"25","author":"D Binns","year":"2009","unstructured":"Binns D, Dimmer E, Huntley R, Barrell D, Odonovan C, Apweiler R. Quickgo: a web-based tool for gene ontology searching. Bioinformatics. 2009;25(22):3045\u20136.","journal-title":"Bioinformatics"},{"key":"4479_CR36","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/978-1-4939-3743-1_12","volume":"1446","author":"C Pesquita","year":"2017","unstructured":"Pesquita C. Semantic similarity in the gene ontology. Methods Mol Biol (Clifton, NJ). 2017;1446:161.","journal-title":"Methods Mol Biol (Clifton, NJ)"},{"key":"4479_CR37","first-page":"2","volume":"15","author":"S Loria","year":"2018","unstructured":"Loria S. Textblob documentation. Release. 2018;15:2.","journal-title":"Release"},{"key":"4479_CR38","unstructured":"Loshchilov I, Hutter F. Decoupled weight decay regularization. In: International conference on learning representations. 2018."},{"key":"4479_CR39","doi-asserted-by":"crossref","unstructured":"Wolf T, Chaumond J, Debut L, Sanh V, Delangue C, Moi A, Cistac P, Funtowicz M, Davison J, Shleifer S et al. Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 2020. pp. 38\u201345.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"issue":"1","key":"4479_CR40","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1186\/s12911-018-0639-1","volume":"18","author":"M Shardlow","year":"2018","unstructured":"Shardlow M, Batista-Navarro R, Thompson P, Nawaz R, McNaught J, Ananiadou S. Identification of research hypotheses and new knowledge from scientific literature. BMC Med Inform Decis Mak. 2018;18(1):46.","journal-title":"BMC Med Inform Decis Mak"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-021-04479-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-021-04479-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-021-04479-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T13:32:54Z","timestamp":1638279174000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-021-04479-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,25]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["4479"],"URL":"https:\/\/doi.org\/10.1186\/s12859-021-04479-9","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.05.26.445910","asserted-by":"object"}]},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,25]]},"assertion":[{"value":"12 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 November 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"KV is a Section Editor for BMC Bioinformatics. KV reports receiving funding from Elsevier BV and speaking fees from Pfizer. All other authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"565"}}