{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T15:47:58Z","timestamp":1750780078927,"version":"3.37.3"},"reference-count":62,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2018,3,20]],"date-time":"2018-03-20T00:00:00Z","timestamp":1521504000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["SFI\/12\/RC\/2289"],"award-info":[{"award-number":["SFI\/12\/RC\/2289"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,7,19]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Motivation<\/jats:title><jats:p>Searching for precise terms and terminological definitions in the biomedical data space is problematic, as researchers find overlapping, closely related and even equivalent concepts in a single or multiple ontologies. Search engines that retrieve ontological resources often suggest an extensive list of search results for a given input term, which leads to the tedious task of selecting the best-fit ontological resource (class or property) for the input term and reduces user confidence in the retrieval engines. A systematic evaluation of these search engines is necessary to understand their strengths and weaknesses in different search requirements.<\/jats:p><\/jats:sec><jats:sec><jats:title>Result<\/jats:title><jats:p>We have implemented seven comparable Information Retrieval ranking algorithms to search through ontologies and compared them against four search engines for ontologies. Free-text queries have been performed, the outcomes have been judged by experts and the ranking algorithms and search engines have been evaluated against the expert-based ground truth (GT). In addition, we propose a probabilistic GT that is developed automatically to provide deeper insights and confidence to the expert-based GT as well as evaluating a broader range of search queries.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>The main outcome of this work is the identification of key search factors for biomedical ontologies together with search requirements and a set of recommendations that will help biomedical experts and ontology engineers to select the best-suited retrieval mechanism in their search scenarios. We expect that this evaluation will allow researchers and practitioners to apply the current search techniques more reliably and that it will help them to select the right solution for their daily work.<\/jats:p><\/jats:sec><jats:sec><jats:title>Availability<\/jats:title><jats:p>The source code (of seven ranking algorithms), ground truths and experimental results are available at https:\/\/github.com\/danielapoliveira\/bioont-search-benchmark<\/jats:p><\/jats:sec>","DOI":"10.1093\/bib\/bby015","type":"journal-article","created":{"date-parts":[[2018,2,14]],"date-time":"2018-02-14T20:11:24Z","timestamp":1518639084000},"page":"1477-1491","source":"Crossref","is-referenced-by-count":6,"title":["Where to search top-K biomedical ontologies?"],"prefix":"10.1093","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0559-8737","authenticated-orcid":false,"given":"Daniela","family":"Oliveira","sequence":"first","affiliation":[{"name":"Insight Centre for Data Analytics, NUI Galway, Ireland"}]},{"given":"Anila Sahar","family":"Butt","sequence":"additional","affiliation":[{"name":"CSIRO, Canberra, Australia"}]},{"given":"Armin","family":"Haller","sequence":"additional","affiliation":[{"name":"Australian National University, Canberra, Australia"}]},{"given":"Dietrich","family":"Rebholz-Schuhmann","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, NUI Galway, Ireland"}]},{"given":"Ratnesh","family":"Sahay","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, NUI Galway, Ireland"}]}],"member":"286","published-online":{"date-parts":[[2018,3,20]]},"reference":[{"issue":"5","key":"2019100807490168600_bby015-B1","doi-asserted-by":"crossref","first-page":"R46","DOI":"10.1186\/gb-2005-6-5-r46","article-title":"Relations in biomedical ontologies","volume":"6","author":"Smith","year":"2005","journal-title":"Genome Biol"},{"volume-title":"Artificial Intelligence in Medicine","year":"1982","author":"Szolovits","key":"2019100807490168600_bby015-B2"},{"key":"2019100807490168600_bby015-B3","first-page":"67","article-title":"Biomedical ontologies in action: role in knowledge management, data integration and decision support","author":"Bodenreider","year":"2008","journal-title":"Yearb Med Inform"},{"issue":"1","key":"2019100807490168600_bby015-B4","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1038\/75556","article-title":"Gene Ontology: tool for the unification of biology","volume":"25","author":"Ashburner","year":"2000","journal-title":"Nat Genet"},{"issue":"6","key":"2019100807490168600_bby015-B5","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1093\/bib\/bbv011","article-title":"The role of ontologies in biological and biomedical research: a functional perspective","volume":"16","author":"Hoehndorf","year":"2015","journal-title":"Brief Bioinform"},{"key":"2019100807490168600_bby015-B6","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.websem.2012.02.003","article-title":"Emerging practices for mapping and linking life sciences data using RDF\u2014a case series","volume":"14","author":"Marshall","year":"2012","journal-title":"Web Semant"},{"year":"2015","author":"Hu","key":"2019100807490168600_bby015-B7"},{"year":"2014","author":"Butt","key":"2019100807490168600_bby015-B8"},{"year":"2007","author":"Zaragoza","key":"2019100807490168600_bby015-B9"},{"year":"2004","author":"Ding","key":"2019100807490168600_bby015-B10"},{"year":"2007","author":"Tummarello","key":"2019100807490168600_bby015-B11"},{"issue":"1","key":"2019100807490168600_bby015-B12","doi-asserted-by":"crossref","first-page":"55","DOI":"10.3233\/SW-2011-0031","article-title":"Watson, more than a semantic web search engine","volume":"2","author":"d\u2019Aquin","year":"2011","journal-title":"Semantic Web"},{"author":"Harth","key":"2019100807490168600_bby015-B13"},{"year":"1999","author":"Page","key":"2019100807490168600_bby015-B14"},{"issue":"4","key":"2019100807490168600_bby015-B15","doi-asserted-by":"crossref","first-page":"810","DOI":"10.1109\/TSMCA.2011.2132705","article-title":"Falcons concept search: a practical search engine for web ontologies","volume":"41","author":"Qu","year":"2011","journal-title":"IEEE Trans Syst Man Cybern A Syst Hum"},{"year":"2005","author":"Gangemi","key":"2019100807490168600_bby015-B16"},{"issue":"2","key":"2019100807490168600_bby015-B17","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1145\/503124.503150","article-title":"Evaluating ontological decisions with OntoClean","volume":"45","author":"Guarino","year":"2002","journal-title":"Commun ACM"},{"issue":"2","key":"2019100807490168600_bby015-B18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4018\/jdm.2004040101","article-title":"ONTOMETRIC: a method to choose the appropriate ontology","volume":"15","author":"Lozano-Tello","year":"2004","journal-title":"J Database Manag"},{"key":"2019100807490168600_bby015-B19","first-page":"1","volume-title":"The Semantic Web - ISWC 2006","author":"Alani","year":"2006"},{"year":"2007","author":"Suchanek","key":"2019100807490168600_bby015-B20"},{"year":"2003","author":"Patel","key":"2019100807490168600_bby015-B21"},{"year":"2007","author":"Thomas","key":"2019100807490168600_bby015-B22"},{"year":"2004","author":"Buitelaar","key":"2019100807490168600_bby015-B23"},{"year":"2007","author":"Buitelaar","key":"2019100807490168600_bby015-B24"},{"issue":"suppl","key":"2019100807490168600_bby015-B25","doi-asserted-by":"crossref","first-page":"W541","DOI":"10.1093\/nar\/gkr469","article-title":"BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications","volume":"39","author":"Whetzel","year":"2011","journal-title":"Nucleic Acids Res"},{"year":"2015","author":"Jupp","key":"2019100807490168600_bby015-B26"},{"issue":"1","key":"2019100807490168600_bby015-B27","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1186\/1471-2105-12-218","article-title":"OntoCAT\u2013simple ontology search and integration in Java, R and REST\/JavaScript","volume":"12","author":"Adamusiak","year":"2011","journal-title":"BMC Bioinformatics"},{"issue":"17","key":"2019100807490168600_bby015-B28","doi-asserted-by":"crossref","first-page":"2468","DOI":"10.1093\/bioinformatics\/btr375","article-title":"ontocat: an R package for ontology traversal and search","volume":"27","author":"Kurbatova","year":"2011","journal-title":"Bioinformatics"},{"issue":"D1","key":"2019100807490168600_bby015-B29","doi-asserted-by":"crossref","first-page":"D926","DOI":"10.1093\/nar\/gkt1270","article-title":"Expression atlas update-a database of gene and transcript expression from microarray-and sequencing-based functional genomics experiments","volume":"42","author":"Petryszak","year":"2014","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2019100807490168600_bby015-B30","doi-asserted-by":"crossref","first-page":"D896","DOI":"10.1093\/nar\/gkw1133","article-title":"The new NHGRI-EBI catalog of published genome-wide association studies (GWAS catalog)","volume":"45","author":"MacArthur","year":"2017","journal-title":"Nucleic Acids Res"},{"issue":"5","key":"2019100807490168600_bby015-B31","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","article-title":"Term-weighting approaches in automatic text retrieval","volume":"24","author":"Salton","year":"1988","journal-title":"Inf Process Manag"},{"year":"1995","author":"Robertson","key":"2019100807490168600_bby015-B32"},{"issue":"11","key":"2019100807490168600_bby015-B33","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1145\/361219.361220","article-title":"A vector space model for automatic indexing","volume":"18","author":"Salton","year":"1975","journal-title":"Commun ACM"},{"year":"1999","author":"Page","key":"2019100807490168600_bby015-B34"},{"issue":"1","key":"2019100807490168600_bby015-B35","doi-asserted-by":"crossref","first-page":"23","DOI":"10.3233\/AO-2011-0087","article-title":"MIREOT: the minimum information to reference an external ontology term","volume":"6","author":"Courtot","year":"2011","journal-title":"Appl Ontol"},{"issue":"1","key":"2019100807490168600_bby015-B36","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3233\/SW-2011-0025","article-title":"The owl api: a java API for owl ontologies","volume":"2","author":"Horridge","year":"2011","journal-title":"Semant Web"},{"issue":"D1","key":"2019100807490168600_bby015-B37","doi-asserted-by":"crossref","first-page":"D456","DOI":"10.1093\/nar\/gks1146","article-title":"The ChEBI reference database and ontology for biologically relevant chemistry: enhancements for 2013","volume":"41","author":"Hastings","year":"2013","journal-title":"Nucleic Acids Res"},{"issue":"2","key":"2019100807490168600_bby015-B38","doi-asserted-by":"crossref","first-page":"R21.","DOI":"10.1186\/gb-2005-6-2-r21","article-title":"An ontology for cell types","volume":"6","author":"Bard","year":"2005","journal-title":"Genome Biol"},{"issue":"D1","key":"2019100807490168600_bby015-B39","doi-asserted-by":"crossref","first-page":"D940","DOI":"10.1093\/nar\/gkr972","article-title":"Disease Ontology: a backbone for disease semantic integration","volume":"40","author":"Schriml","year":"2012","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"2019100807490168600_bby015-B40","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1186\/2041-1480-4-44","article-title":"Building a drug ontology based on rxnorm and other sources","volume":"4","author":"Hanna","year":"2013","journal-title":"J Biomed Semant"},{"issue":"10","key":"2019100807490168600_bby015-B41","doi-asserted-by":"crossref","first-page":"1325","DOI":"10.1093\/bioinformatics\/btt113","article-title":"EDAM: an ontology of bioinformatics operations, types of data and identifiers, topics and formats","volume":"29","author":"Ison","year":"2013","journal-title":"Bioinformatics"},{"issue":"8","key":"2019100807490168600_bby015-B42","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1093\/bioinformatics\/btq099","article-title":"Modeling sample variables with an experimental factor ontology","volume":"26","author":"Malone","year":"2010","journal-title":"Bioinformatics"},{"issue":"6","key":"2019100807490168600_bby015-B43","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1016\/j.jbi.2003.11.007","article-title":"A reference ontology for biomedical informatics: the foundational model of anatomy","volume":"36","author":"Rosse","year":"2003","journal-title":"J Biomed Inform"},{"issue":"D1","key":"2019100807490168600_bby015-B44","doi-asserted-by":"crossref","first-page":"D966","DOI":"10.1093\/nar\/gkt1026","article-title":"The human phenotype ontology project: linking molecular biology and disease through phenotype data","volume":"42","author":"K\u00f6hler","year":"2013","journal-title":"Nucleic Acids Res"},{"issue":"9\u201310","key":"2019100807490168600_bby015-B45","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1007\/s00335-015-9584-9","article-title":"Mouse anatomy ontologies: enhancements and tools for exploring and integrating biomedical data","volume":"26","author":"Hayamizu","year":"2015","journal-title":"Mamm Genome"},{"issue":"1","key":"2019100807490168600_bby015-B46","doi-asserted-by":"crossref","first-page":"R7","DOI":"10.1186\/gb-2004-6-1-r7","article-title":"The Mammalian Phenotype Ontology as a tool for annotating, analyzing and comparing phenotypic information","volume":"6","author":"Smith","year":"2004","journal-title":"Genome Biol"},{"issue":"1","key":"2019100807490168600_bby015-B47","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/2041-1480-4-18","article-title":"The mouse pathology ontology, MPATH; structure and applications","volume":"4","author":"Schofield","year":"2013","journal-title":"J Biomed Semant"},{"key":"2019100807490168600_bby015-B48","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/B978-0-12-388408-4.00004-6","article-title":"The neurobehavior ontology: an ontology for annotation and integration of behavior and behavioral phenotypes","volume":"103","author":"Gkoutos","year":"2012","journal-title":"Int Rev Neurobiol"},{"issue":"1","key":"2019100807490168600_bby015-B49","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.jbi.2006.02.013","article-title":"NCI Thesaurus: a semantic model integrating cancer-related clinical and molecular information","volume":"40","author":"Sioutos","year":"2007","journal-title":"J Biomed Inform"},{"key":"2019100807490168600_bby015-B50","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1186\/2041-1480-5-29","article-title":"OAE: the ontology of adverse events","volume":"5","author":"He","year":"2014","journal-title":"J Biomed Semant"},{"year":"2014","author":"He","key":"2019100807490168600_bby015-B51"},{"issue":"1","key":"2019100807490168600_bby015-B52","doi-asserted-by":"crossref","first-page":"R2","DOI":"10.1186\/gb-2010-11-1-r2","article-title":"Integrating phenotype ontologies across multiple species","volume":"11","author":"Mungall","year":"2010","journal-title":"Genome Biol"},{"key":"2019100807490168600_bby015-B53","doi-asserted-by":"crossref","first-page":"D449","DOI":"10.1093\/nar\/gkm908","article-title":"The Plant Ontology Database: a community resource for plant structure and developmental stages controlled vocabulary and annotations","volume":"36","author":"Avraham","year":"2008","journal-title":"Nucleic Acids Res"},{"author":"Haendel","key":"2019100807490168600_bby015-B54"},{"key":"2019100807490168600_bby015-B55","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1186\/2041-1480-4-13","article-title":"The Vertebrate Trait Ontology: a controlled vocabulary for the annotation of trait data across species","volume":"4","author":"Park","year":"2013","journal-title":"J Biomed Semant"},{"issue":"1","key":"2019100807490168600_bby015-B56","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1186\/1471-2105-12-32","article-title":"Worm Phenotype Ontology: integrating phenotype data within and beyond the C. elegans community","volume":"12","author":"Schindelman","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2019100807490168600_bby015-B57","doi-asserted-by":"crossref","first-page":"92.","DOI":"10.1186\/1471-213X-8-92","article-title":"An ontology for Xenopus anatomy and development","volume":"8","author":"Segerdell","year":"2008","journal-title":"BMC Dev Biol"},{"key":"2019100807490168600_bby015-B58","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1186\/2041-1480-5-12","article-title":"The zebrafish anatomy and stage ontologies: representing the anatomy and development of Danio rerio","volume":"5","author":"Van Slyke","year":"2014","journal-title":"J Biomed Semant"},{"issue":"3","key":"2019100807490168600_bby015-B59","doi-asserted-by":"crossref","first-page":"1621","DOI":"10.1214\/aos\/1176349275","article-title":"Chi-square goodness-of-fit tests for randomly censored data","volume":"21","author":"Kim","year":"1993","journal-title":"Ann Stat"},{"key":"2019100807490168600_bby015-B60","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/978-3-642-36824-0_15","volume-title":"Graphics Recognition. New Trends and Challenges","author":"Lamiroy","year":"2013"},{"year":"2013","author":"Wang","key":"2019100807490168600_bby015-B61"},{"author":"Gavankar","key":"2019100807490168600_bby015-B62"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/20\/4\/1477\/30119637\/bby015.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/20\/4\/1477\/30119637\/bby015.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,10,28]],"date-time":"2020-10-28T04:52:02Z","timestamp":1603860722000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/20\/4\/1477\/4944514"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,3,20]]},"references-count":62,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2018,3,20]]},"published-print":{"date-parts":[[2019,7,19]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bby015","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"type":"print","value":"1467-5463"},{"type":"electronic","value":"1477-4054"}],"subject":[],"published-other":{"date-parts":[[2019,7]]},"published":{"date-parts":[[2018,3,20]]}}}