{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T00:09:52Z","timestamp":1773965392722,"version":"3.50.1"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2019,2,4]],"date-time":"2019-02-04T00:00:00Z","timestamp":1549238400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Spanish Ministry of Economy, Industry and Competitiveness"},{"DOI":"10.13039\/501100008530","name":"European Regional Development Fund","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100008530","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007801","name":"Fundaci\u00f3n S\u00e9neca","doi-asserted-by":"publisher","award":["TIN2014-53749-C2-2-R"],"award-info":[{"award-number":["TIN2014-53749-C2-2-R"]}],"id":[{"id":"10.13039\/100007801","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007801","name":"Fundaci\u00f3n S\u00e9neca","doi-asserted-by":"publisher","award":["TIN2017-85949-C2-1-R"],"award-info":[{"award-number":["TIN2017-85949-C2-1-R"]}],"id":[{"id":"10.13039\/100007801","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007801","name":"Fundaci\u00f3n S\u00e9neca","doi-asserted-by":"publisher","award":["19371\/PI\/14"],"award-info":[{"award-number":["19371\/PI\/14"]}],"id":[{"id":"10.13039\/100007801","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,3,23]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The development and application of biological ontologies have increased significantly in recent years. These ontologies can be retrieved from different repositories, which do not provide much information about quality aspects of the ontologies. In the past years, some ontology structural metrics have been proposed, but their validity as measurement instrument has not been sufficiently studied to date. In this work, we evaluate a set of reproducible and objective ontology structural metrics. Given the lack of standard methods for this purpose, we have applied an evaluation method based on the stability and goodness of the classifications of ontologies produced by each metric on an ontology corpus. The evaluation has been done using ontology repositories as corpora. More concretely, we have used 119 ontologies from the OBO Foundry repository and 78 ontologies from AgroPortal. First, we study the correlations between the metrics. Second, we study whether the clusters for a given metric are stable and have a good structure. The results show that the existing correlations are not biasing the evaluation, there are no metrics generating unstable clusterings and all the metrics evaluated provide at least reasonable clustering structure. Furthermore, our work permits to review and suggest the most reliable ontology structural metrics in terms of stability and goodness of their classifications.<\/jats:p>\n               <jats:p>Availability: http:\/\/sele.inf.um.es\/ontology-metrics<\/jats:p>","DOI":"10.1093\/bib\/bbz009","type":"journal-article","created":{"date-parts":[[2019,1,9]],"date-time":"2019-01-09T02:50:32Z","timestamp":1547002232000},"page":"473-485","source":"Crossref","is-referenced-by-count":11,"title":["Evaluation of ontology structural metrics based on public repository data"],"prefix":"10.1093","volume":"21","author":[{"given":"Manuel","family":"Franco","sequence":"first","affiliation":[{"name":"Departamento de Estad\u00edstica e Investigaci\u00f3n Operativa, Universidad de Murcia, Murcia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juana Mar\u00eda","family":"Vivo","sequence":"additional","affiliation":[{"name":"Departamento de Estad\u00edstica e Investigaci\u00f3n Operativa, Universidad de Murcia, Murcia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Manuel","family":"Quesada-Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Center of Operations Research (CIO), Miguel Hern\u00e1ndez University of Elche, Elche, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Astrid","family":"Duque-Ramos","sequence":"additional","affiliation":[{"name":"Departamento de Sistemas, Facultad de Ingenier\u00edas, Universidad de Antioquia, Medell\u00edn, Colombia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7558-2880","authenticated-orcid":false,"given":"Jesualdo Tom\u00e1s","family":"Fern\u00e1ndez-Breis","sequence":"additional","affiliation":[{"name":"Departamento de Inform\u00e1tica y Sistemas, Universidad de Murcia, IMIB-Arrixaca, Murcia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2019,2,4]]},"reference":[{"key":"2020080709363122700_ref1","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/j.knosys.2016.05.016","article-title":"A semantic web based framework for the interoperability and exploitation of clinical models and EHR data","volume":"105","author":"Legaz-Garc\u00eda","year":"2016","journal-title":"Knowl Based Syst"},{"issue":"1","key":"2020080709363122700_ref2","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1093\/bib\/bbm059","article-title":"Biomedical ontologies: a functional perspective","volume":"9","author":"Rubin","year":"2007","journal-title":"Brief Bioinformatics"},{"key":"2020080709363122700_ref3","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.knosys.2016.08.002","article-title":"Human endocrine system modeling based on ontologies","volume":"111","author":"Viale","year":"2016","journal-title":"Knowl Based Syst"},{"issue":"6","key":"2020080709363122700_ref4","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1093\/bib\/bbv011","article-title":"The role of ontologies in biological and biomedical research: a functional perspective","volume":"16","author":"Hoehndorf","year":"2015","journal-title":"Brief Bioinformatics"},{"issue":"11","key":"2020080709363122700_ref5","doi-asserted-by":"crossref","first-page":"1251","DOI":"10.1038\/nbt1346","article-title":"The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration","volume":"25","author":"Smith","year":"2007","journal-title":"Nat Biotechnol"},{"key":"2020080709363122700_ref6","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.compag.2017.10.012","article-title":"AgroPortal: a vocabulary and ontology repository for agronomy","volume":"144","author":"Jonquet","year":"2018","journal-title":"Comput Electron Agri"},{"issue":"D1","key":"2020080709363122700_ref7","doi-asserted-by":"crossref","first-page":"D347","DOI":"10.1093\/nar\/gkw918","article-title":"Ontobee: a linked ontology data server to support ontology term dereferencing, linkage, query and integration","volume":"45","author":"Ong","year":"2017","journal-title":"Nucleic Acids Res"},{"issue":"suppl 2","key":"2020080709363122700_ref8","doi-asserted-by":"crossref","first-page":"W155","DOI":"10.1093\/nar\/gkq331","article-title":"The ontology lookup service: bigger and better","volume":"38","author":"C\u00f4t\u00e9","year":"2010","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"2020080709363122700_ref9","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1186\/s12859-015-0456-9","article-title":"Aber-owl: a framework for ontology-based data access in biology","volume":"16","author":"Hoehndorf","year":"2015","journal-title":"BMC Bioinformatics"},{"issue":"suppl 2","key":"2020080709363122700_ref10","doi-asserted-by":"crossref","first-page":"W541","DOI":"10.1093\/nar\/gkr469","article-title":"BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications","volume":"39","author":"Whetzel","year":"2011","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"2020080709363122700_ref11","doi-asserted-by":"crossref","first-page":"179","DOI":"10.3233\/AO-130125","article-title":"Towards ontology evaluation across the life cycle","volume":"8","author":"Neuhaus","year":"2013","journal-title":"Appl Ontol"},{"issue":"6","key":"2020080709363122700_ref12","doi-asserted-by":"crossref","first-page":"696","DOI":"10.1093\/bib\/bbs053","article-title":"Evaluation of research in biomedical ontologies","volume":"14","author":"Hoehndorf","year":"2012","journal-title":"Brief Bioinformatics"},{"key":"2020080709363122700_ref13","first-page":"140","volume-title":"The Semantic Web: Research and Applications","author":"Gangemi"},{"issue":"3","key":"2020080709363122700_ref14","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1055\/s-0038-1634078","article-title":"Quality assurance of medical ontologies","volume":"45","author":"Rogers","year":"2006","journal-title":"Methods Inf Med"},{"issue":"3","key":"2020080709363122700_ref15","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1016\/j.jbi.2008.12.008","article-title":"Applying evolutionary terminology auditing to the gene ontology","volume":"42","author":"Ceusters","year":"2009","journal-title":"J Biomed Inf"},{"key":"2020080709363122700_ref16","first-page":"96","article-title":"Applying evolutionary terminology auditing to SNOMED CT","volume-title":"AMIA Annual Symposium Proceedings, 2010.","author":"Ceusters"},{"issue":"2","key":"2020080709363122700_ref17","first-page":"159","article-title":"OQuaRE: a SQuaRE-based approach for evaluating the quality of ontologies","volume":"43","author":"Duque-Ramos","year":"2011","journal-title":"J Res Prac Inf Tech"},{"key":"2020080709363122700_ref18","first-page":"185","volume-title":"ICSC \u201807: Proceedings of the International Conference on Semantic Computing, 2007","author":"Tartir"},{"issue":"1","key":"2020080709363122700_ref19","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3844\/jcssp.2005.107.113","article-title":"Cohesion metrics for ontology design and application","volume":"1","author":"Yao","year":"2005","journal-title":"J Comp Sci"},{"key":"2020080709363122700_ref20","first-page":"121","volume-title":"AMIA Annual Symposium Proceedings, 2006","author":"Ceusters"},{"key":"2020080709363122700_ref21","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.knosys.2015.02.026","article-title":"Ontology usage analysis in the ontology lifecycle: a state-of-the-art review","volume":"80","author":"Ashraf","year":"2015","journal-title":"Knowl Based Syst"},{"key":"2020080709363122700_ref22","first-page":"24","volume-title":"International Conference on Applications of Natural Language to Information Systems","author":"McDaniel","year":"2016"},{"issue":"2","key":"2020080709363122700_ref23","doi-asserted-by":"crossref","first-page":"7","DOI":"10.4018\/ijswis.2014040102","article-title":"Oops!(Ontology Pitfall Scanner!): an on-line tool for ontology evaluation","volume":"10","author":"Poveda-Villal\u00f3n","year":"2014","journal-title":"Int J Semant Web Inf Syst"},{"issue":"140","key":"2020080709363122700_ref24","first-page":"1","article-title":"A technique for the measurement of attitudes","volume":"22","author":"Likert","year":"1932","journal-title":"Arch Psychol"},{"issue":"6","key":"2020080709363122700_ref25","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1109\/32.295895","article-title":"A metrics suite for object oriented design","volume":"20","author":"Chidamber","year":"1994","journal-title":"IEEE Trans Softw Eng"},{"issue":"2","key":"2020080709363122700_ref26","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/S0164-1212(98)10052-3","article-title":"Another metric suite for object-oriented programming","volume":"44","author":"Li","year":"1998","journal-title":"J Syst Softw"},{"key":"2020080709363122700_ref27","doi-asserted-by":"crossref","first-page":"180118","DOI":"10.1038\/sdata.2018.118","article-title":"A design framework and exemplar metrics for FAIRness","volume":"5","author":"Wilkinson","year":"2018","journal-title":"Sci Data"},{"issue":"1","key":"2020080709363122700_ref28","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1186\/s13326-016-0091-z","article-title":"Supporting the analysis of ontology evolution processes through the combination of static and dynamic scaling functions in oquare","volume":"7","author":"Duque-Ramos","year":"2016","journal-title":"J Biomed Semant"},{"key":"2020080709363122700_ref29","first-page":"426","article-title":"Preliminary analysis of the OBO Foundry ontologies and their evolution using OQuaRE","volume":"235","author":"Quesada-Mart\u00ednez","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"2020080709363122700_ref30","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/BF01246105","article-title":"Measuring the influence of individual data points in a cluster analysis","volume":"13","author":"Cheng","year":"1996","journal-title":"J Classification"},{"key":"2020080709363122700_ref31","first-page":"241","article-title":"Distribution de la flore alpine dans le Basin de Dranses et dans quelques regions voisines","volume":"37","author":"Jaccard","year":"1901","journal-title":"Bulletin de la Societe Vaudoise des Sciences Naturelles"},{"key":"2020080709363122700_ref32","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.csda.2006.11.025","article-title":"Cluster-wise assessment of cluster stability","volume":"52","author":"Hennig","year":"2007","journal-title":"Comput Stat Data Anal"},{"key":"2020080709363122700_ref33","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: a graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J Comput Appl Math"},{"issue":"1","key":"2020080709363122700_ref34","first-page":"1","article-title":"A dendrite method for cluster analysis","volume":"3","author":"Calinski","year":"1974","journal-title":"Commun Stat"},{"issue":"1","key":"2020080709363122700_ref35","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1080\/01969727408546059","article-title":"Well-separated clusters and optimal fuzzy partitions","volume":"4","author":"Dunn","year":"1974","journal-title":"J Cybern"},{"issue":"2","key":"2020080709363122700_ref36","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/TPAMI.1979.4766909","article-title":"A cluster separation measure","volume":"PAMI-1","author":"Davies","year":"1979","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2020080709363122700_ref37","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.ins.2017.02.010","article-title":"Using the stability of objects to determine the number of clusters in datasets","volume":"393","author":"Lord","year":"2017","journal-title":"Inf Sci"},{"key":"2020080709363122700_ref38","doi-asserted-by":"crossref","DOI":"10.1002\/9780470316801","volume-title":"Finding Groups in Data: An Introduction to Cluster Analysis","author":"Kaufman","year":"1990"},{"issue":"1","key":"2020080709363122700_ref39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3233\/SW-2011-0025","article-title":"The OWL API: a Java API for OWL ontologies","volume":"2","author":"Horridge","year":"2011","journal-title":"Semantic Web"},{"key":"2020080709363122700_ref40","volume-title":"Proceedings of the 1st International Workshop on OWL Reasoner Evaluation,","author":"Kazakov","year":"2012"},{"key":"2020080709363122700_ref41","first-page":"91","volume-title":"OWLED,","author":"Shearer","year":"2008"},{"key":"2020080709363122700_ref42","volume-title":"R language definition.","author":"Team","year":"2000"},{"key":"2020080709363122700_ref43","author":"Wei","year":"2016"},{"key":"2020080709363122700_ref44","author":"Hennig","year":"2015"},{"key":"2020080709363122700_ref45","author":"Maechler","year":"2017"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/21\/2\/473\/33585241\/bbz009.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/21\/2\/473\/33585241\/bbz009.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,8,7]],"date-time":"2020-08-07T13:51:39Z","timestamp":1596808299000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/21\/2\/473\/5304005"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,2,4]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2019,2,4]]},"published-print":{"date-parts":[[2020,3,23]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbz009","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,3]]},"published":{"date-parts":[[2019,2,4]]}}}