{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T10:56:29Z","timestamp":1740135389872,"version":"3.37.3"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"S10","license":[{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,12,15]],"date-time":"2020-12-15T00:00:00Z","timestamp":1607990400000},"content-version":"vor","delay-in-days":14,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Ontologies house various kinds of domain knowledge in formal structures, primarily in the form of concepts and the associative relationships between them. Ontologies have become integral components of many health information processing environments. Hence, quality assurance of the conceptual content of any ontology is critical. Relationships are foundational to the definition of concepts.<jats:italic>Missing relationship errors<\/jats:italic>(i.e., unintended omissions of important definitional relationships) can have a deleterious effect on the quality of an ontology. An abstraction network is a structure that overlays an ontology and provides an alternate, summarization view of its contents. One kind of abstraction network is called an<jats:italic>area taxonomy<\/jats:italic>, and a variation of it is called a<jats:italic>subtaxonomy<\/jats:italic>. A methodology based on these taxonomies for more readily finding missing relationship errors is explored.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The<jats:italic>area taxonomy<\/jats:italic>and the<jats:italic>subtaxonomy<\/jats:italic>are deployed to help reveal concepts that have a high likelihood of exhibiting missing relationship errors. A specific top-level grouping unit found within the area taxonomy and subtaxonomy, when deemed to be anomalous, is used as an indicator that missing relationship errors are likely to be found among certain concepts. Two hypotheses pertaining to the effectiveness of our Quality Assurance approach are studied.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Our Quality Assurance methodology was applied to the<jats:italic>Biological Process<\/jats:italic>hierarchy of the National Cancer Institute thesaurus (NCIt) and SNOMED CT\u2019s<jats:italic>Eye\/vision finding<\/jats:italic>subhierarchy within its<jats:italic>Clinical finding<\/jats:italic>hierarchy. Many missing relationship errors were discovered and confirmed in our analysis. For both test-bed hierarchies, our Quality Assurance methodology yielded a statistically significantly higher number of concepts with missing relationship errors in comparison to a control sample of concepts. Two hypotheses are confirmed by these findings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Quality assurance is a critical part of an ontology\u2019s lifecycle, and automated or semi-automated tools for supporting this process are invaluable. We introduced a Quality Assurance methodology targeted at missing relationship errors. Its successful application to the NCIt\u2019s<jats:italic>Biological Process<\/jats:italic>hierarchy and SNOMED CT\u2019s<jats:italic>Eye\/vision finding<\/jats:italic>subhierarchy indicates that it can be a useful addition to the arsenal of tools available to ontology maintenance personnel.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-020-01319-3","type":"journal-article","created":{"date-parts":[[2020,12,15]],"date-time":"2020-12-15T07:02:46Z","timestamp":1608015766000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Missing lateral relationships in top-level concepts of an ontology"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6709-828X","authenticated-orcid":false,"given":"Ling","family":"Zheng","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hua","family":"Min","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"P. Lloyd","family":"Hildebrand","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Halper","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Geller","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sherri","family":"de Coronado","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yehoshua","family":"Perl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,12,15]]},"reference":[{"key":"1319_CR1","first-page":"7","volume":"5","author":"K Giannangelo","year":"2008","unstructured":"Giannangelo K, Fenton SH. SNOMED CT survey: an assessment of implementation in EMR\/EHR applications. Perspect Health Inf Manag. 2008;5:7.","journal-title":"Perspect Health Inf Manag"},{"key":"1319_CR2","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/S0084-3873(08)79094-1","volume":"2008","author":"O Bodenreider","year":"2008","unstructured":"Bodenreider O. Biomedical ontologies in action: role in knowledge management, data integration and decision support. Yearb Med Inform. 2008;2008:67\u201379.","journal-title":"Yearb Med Inform."},{"issue":"6","key":"1319_CR3","doi-asserted-by":"publisher","first-page":"1069","DOI":"10.1093\/bib\/bbv011","volume":"16","author":"R Hoehndorf","year":"2015","unstructured":"Hoehndorf R, Schofield PN, Gkoutos GV. The role of ontologies in biological and biomedical research: a functional perspective. Brief Bioinform. 2015;16(6):1069\u201380.","journal-title":"Brief Bioinform"},{"key":"1319_CR4","unstructured":"Ochs C, Case JT, Perl Y. Tracking the remodeling of SNOMED CT's bacterial infectious diseases. In: AMIA annual symposium proceeding, vol 2016; 2016. p. 974\u201383."},{"issue":"1","key":"1319_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artmed.2015.03.005","volume":"64","author":"M Halper","year":"2015","unstructured":"Halper M, Gu H, Perl Y, Ochs C. Abstraction networks for terminologies: supporting management of \u201cbig knowledge.\u201d Artif Intell Med. 2015;64(1):1\u201316.","journal-title":"Artif Intell Med"},{"issue":"6","key":"1319_CR6","doi-asserted-by":"publisher","first-page":"676","DOI":"10.1197\/jamia.M2036","volume":"13","author":"H Min","year":"2006","unstructured":"Min H, Perl Y, Chen Y, Halper M, Geller J, Wang Y. Auditing as part of the terminology design life cycle. J Am Med Inform Assoc. 2006;13(6):676\u201390.","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"1319_CR7","doi-asserted-by":"publisher","first-page":"1642001","DOI":"10.1142\/S0219720016420014","volume":"14","author":"C Ochs","year":"2016","unstructured":"Ochs C, Perl Y, Halper M, Geller J, Lomax J. Quality assurance of the gene ontology using abstraction networks. J Bioinform Comput Biol. 2016;14(3):1642001.","journal-title":"J Bioinform Comput Biol"},{"issue":"5","key":"1319_CR8","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1016\/j.jbi.2006.12.003","volume":"40","author":"Y Wang","year":"2007","unstructured":"Wang Y, Halper M, Min H, Perl Y, Chen Y, Spackman KA. Structural methodologies for auditing SNOMED. J Biomed Inform. 2007;40(5):561\u201381.","journal-title":"J Biomed Inform"},{"issue":"Pt 1","key":"1319_CR9","first-page":"33","volume":"107","author":"S de Coronado","year":"2004","unstructured":"de Coronado S, Haber MW, Sioutos N, Tuttle MS, Wright LW. NCI Thesaurus: using science-based terminology to integrate cancer research results. Stud Health Technol Inform. 2004;107(Pt 1):33\u20137.","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"1319_CR10","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1038\/75556","volume":"25","author":"M Ashburner","year":"2000","unstructured":"Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, et al. Gene ontology: tool for the unification of biology. Gene Ontol Consort Nat Genet. 2000;25(1):25\u20139.","journal-title":"Gene Ontol Consort Nat Genet"},{"key":"1319_CR11","unstructured":"Stearns MQ, Price C, Spackman KA, Wang AY. SNOMED clinical terms: overview of the development process and project status. In: Proceedings of AMIA annual symposium. 2001. p. 662\u20136."},{"issue":"3","key":"1319_CR12","doi-asserted-by":"publisher","first-page":"199","DOI":"10.3233\/AO-190211","volume":"14","author":"H Yumak","year":"2019","unstructured":"Yumak H, Zheng L, Chen L, Halper M, Perl Y, Owen G. Quality assurance of complex ChEBI concepts based on number of relationship types. Appl Ontol. 2019;14(3):199\u2013214.","journal-title":"Appl Ontol"},{"key":"1319_CR13","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.artmed.2017.05.002","volume":"79","author":"G Elhanan","year":"2017","unstructured":"Elhanan G, Ochs C, Mejino JLV Jr, Liu H, Mungall CJ, Perl Y. From SNOMED CT to Uberon: transferability of evaluation methodology between similarly structured ontologies. Artif Intell Med. 2017;79:9\u201314.","journal-title":"Artif Intell Med"},{"key":"1319_CR14","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1016\/j.jbi.2017.07.013","volume":"73","author":"L Zheng","year":"2017","unstructured":"Zheng L, Yumak H, Chen L, Ochs C, Geller J, Kapusnik-Uner J, et al. Quality assurance of chemical ingredient classification for the National Drug File-Reference Terminology. J Biomed Inform. 2017;73:30\u201342.","journal-title":"J Biomed Inform"},{"key":"1319_CR15","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.jbi.2013.11.002","volume":"52","author":"I Sim","year":"2014","unstructured":"Sim I, Tu SW, Carini S, Lehmann HP, Pollock BH, Peleg M, et al. The Ontology of Clinical Research (OCRe): an informatics foundation for the science of clinical research. J Biomed Inform. 2014;52:78\u201391.","journal-title":"J Biomed Inform"},{"key":"1319_CR16","unstructured":"Arabandi S, Ogbuji C, Redline S, Chervin R, Boero J, Benca R, Zhang GQ. Developing a sleep domain ontology. In: AMIA summit on clinical research informatics. 2010. p. 83."},{"issue":"3","key":"1319_CR17","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1515\/jib-2010-126","volume":"7","author":"D Qi","year":"2010","unstructured":"Qi D, King RD, Hopkins AL, Bickerton GR, Soldatova LN. An ontology for description of drug discovery investigations. J Integr Bioinform. 2010;7(3):126.","journal-title":"J Integr Bioinform."},{"issue":"2","key":"1319_CR18","doi-asserted-by":"publisher","first-page":"127","DOI":"10.3233\/SW-130112","volume":"5","author":"D Zeginis","year":"2013","unstructured":"Zeginis D, Hasnain A, Loutas N, Deus HF, Fox R, Tarabanis K. A collaborative methodology for developing a semantic model for interlinking Cancer Chemoprevention linked-data sources. Sementic Web. 2013;5(2):127\u201342.","journal-title":"Sementic Web"},{"key":"1319_CR19","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.jbi.2016.06.008","volume":"62","author":"C Ochs","year":"2016","unstructured":"Ochs C, Geller J, Perl Y, Musen MA. A unified software framework for deriving, visualizing, and exploring abstraction networks for ontologies. J Biomed Inform. 2016;62:90\u2013105.","journal-title":"J Biomed Inform"},{"issue":"Web Server issu","key":"1319_CR20","doi-asserted-by":"publisher","first-page":"W541","DOI":"10.1093\/nar\/gkr469","volume":"39","author":"PL Whetzel","year":"2011","unstructured":"Whetzel PL, Noy NF, Sham NH, Alexander PR, Nyulas C, Tudorache T, et al. BioPortal: enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications. Nucleic Acids Res: NAR. 2011;39(Web Server issue):W541\u20135.","journal-title":"Nucleic Acids Res: NAR"},{"key":"1319_CR21","unstructured":"Halper M, Wang Y, Min H, Chen Y, Hripcsak G, Perl Y, et al. Analysis of error concentrations in SNOMED. In: AMIA annual symposium proceedings. 2007. p. 314\u20138."},{"key":"1319_CR22","unstructured":"Wang Y, Wei D, Xu J, Elhanan G, Perl Y, Halper M, et al. Auditing complex concepts in overlapping subsets of SNOMED. In: AMIA annual symposium proceedings. 2008. p. 273\u20137."},{"key":"1319_CR23","doi-asserted-by":"crossref","unstructured":"Zheng L, Chen Y, Perl Y, Halper M, Geller J, Coronado SD. Quality assurance of concept roles in the National Cancer Institute thesaurus. The 1st international workshop on quality assurance of biological and biomedical ontologies and terminologies in BIBM. 2018. p. 2001\u20138.","DOI":"10.1109\/BIBM.2018.8621277"},{"issue":"4","key":"1319_CR24","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/2757001.2757003","volume":"1","author":"MA Musen","year":"2015","unstructured":"Musen MA, Protege T. The protege project: a look back and a look forward. AI Matters. 2015;1(4):4\u201312.","journal-title":"AI Matters"},{"key":"1319_CR25","first-page":"43","volume-title":"The description logic handbook: theory, implementation, and applications","author":"F Baader","year":"2003","unstructured":"Baader F, Nutt W. Basic description logics. In: Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF, editors. The description logic handbook: theory, implementation, and applications. New York: Cambridge University Press; 2003. p. 43\u201395."},{"key":"1319_CR26","doi-asserted-by":"crossref","unstructured":"Baader F. Restricted role-value-maps in a description logic with existential restrictions and terminological cycles. In: Proceedings of the 2003 international workshop on description logics (DL2003), Rome, Italy September 5\u20137. 2003.","DOI":"10.25368\/2022.125"},{"issue":"3","key":"1319_CR27","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/amiajnl-2014-003151","volume":"22","author":"C Ochs","year":"2014","unstructured":"Ochs C, Geller J, Perl Y, Chen Y, Xu J, Min H, et al. Scalable quality assurance for large SNOMED CT hierarchies using subject-based subtaxonomies. J Am Med Inform Assoc. 2014;22(3):507\u201318.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1319_CR28","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.jbi.2011.08.013","volume":"45","author":"Y Wang","year":"2012","unstructured":"Wang Y, Halper M, Wei D, Perl Y, Geller J. Abstraction of complex concepts with a refined partial-area taxonomy of SNOMED. J Biomed Inform. 2012;45(1):15\u201329.","journal-title":"J Biomed Inform"},{"issue":"1","key":"1319_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jbi.2011.08.016","volume":"45","author":"Y Wang","year":"2012","unstructured":"Wang Y, Halper M, Wei D, Gu H, Perl Y, Xu J, et al. Auditing complex concepts of SNOMED using a refined hierarchical abstraction network. J Biomed Inform. 2012;45(1):1\u201314.","journal-title":"J Biomed Inform"},{"issue":"3","key":"1319_CR30","doi-asserted-by":"publisher","first-page":"200","DOI":"10.3414\/ME16-01-0085","volume":"56","author":"H Min","year":"2017","unstructured":"Min H, Zheng L, Perl Y, Halper M, Coronado SD, Ochs C. Relating complexity and error rates of ontology concepts: More complex NCIt concepts have more errors. Methods Inf Med. 2017;56(3):200\u20138.","journal-title":"Methods Inf Med"},{"key":"1319_CR31","unstructured":"Ochs C, Perl Y, Geller J, Halper M, Gu H, Chen Y, et al. Scalability of abstraction-network-based quality assurance to large SNOMED hierarchies. In: Proceedings of AMIA annual symposium, vol 2013. 2013. p. 1071\u201380."},{"issue":"2","key":"1319_CR32","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1016\/j.ophtha.2007.11.026","volume":"115","author":"HD Hoskins","year":"2008","unstructured":"Hoskins HD, Hildebrand PL, Lum F. The American Academy of Ophthalmology adopts SNOMED CT as its official clinical terminology. Ophthalmology. 2008;115(2):225\u20136.","journal-title":"Ophthalmology"},{"key":"1319_CR33","volume-title":"Data structures and algorithms in Java","author":"MT Goodrich","year":"2014","unstructured":"Goodrich MT, Tamassia R, Goldwasser MH. Data structures and algorithms in Java. Hoboken: Wiley Publishing; 2014."},{"key":"1319_CR34","volume-title":"Permutation, parametric, and bootstrap tests of hypotheses: a practical guide to resampling","author":"PI Good","year":"2005","unstructured":"Good PI. Permutation, parametric, and bootstrap tests of hypotheses: a practical guide to resampling. New York, NY: Springer; 2005."},{"issue":"Suppl 1","key":"1319_CR35","doi-asserted-by":"publisher","first-page":"i36","DOI":"10.1136\/amiajnl-2011-000341","volume":"18","author":"G Elhanan","year":"2011","unstructured":"Elhanan G, Perl Y, Geller J. A survey of SNOMED CT direct users, 2010: impressions and preferences regarding content and quality. J Am Med Inform Assoc. 2011;18(Suppl 1):i36-44.","journal-title":"J Am Med Inform Assoc"},{"issue":"Pt 2","key":"1319_CR36","first-page":"1070","volume":"160","author":"D Wei","year":"2010","unstructured":"Wei D, Bodenreider O. Using the abstraction network in complement to description logics for quality assurance in biomedical terminologies\u2014a case study in SNOMED CT. Stud Health Technol Inform. 2010;160(Pt 2):1070\u20134.","journal-title":"Stud Health Technol Inform"},{"key":"1319_CR37","unstructured":"He Z, Ochs C, Agrawal A, Perl Y, Zeginis D, Tarabanis K, et al. A family-based framework for supporting quality assurance of biomedical ontologies in BioPortal. In: Proceedings of AMIA annual symposium. 2013. p. 581\u201390."},{"key":"1319_CR38","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.jbi.2016.03.007","volume":"61","author":"C Ochs","year":"2016","unstructured":"Ochs C, He Z, Zheng L, Geller J, Perl Y, Hripcsak G, et al. Utilizing a structural meta-ontology for family-based quality assurance of the BioPortal ontologies. J Biomed Inform. 2016;61:63\u201376.","journal-title":"J Biomed Inform"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01319-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-020-01319-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01319-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T00:08:50Z","timestamp":1670198930000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-01319-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":38,"journal-issue":{"issue":"S10","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1319"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-01319-3","relation":{},"ISSN":["1472-6947"],"issn-type":[{"type":"electronic","value":"1472-6947"}],"subject":[],"published":{"date-parts":[[2020,12]]},"assertion":[{"value":"29 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 November 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"305"}}