{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,26]],"date-time":"2025-10-26T22:51:43Z","timestamp":1761519103633},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"3","content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2015,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Objective Standards terminologies may be large and complex, making their quality assurance challenging. Some terminology quality assurance (TQA) methodologies are based on abstraction networks (AbNs), compact terminology summaries. We have tested AbNs and the performance of related TQA methodologies on small terminology hierarchies. However, some standards terminologies, for example, SNOMED, are composed of very large hierarchies. Scaling AbN TQA techniques to such hierarchies poses a significant challenge. We present a scalable subject-based approach for AbN TQA.<\/jats:p>\n               <jats:p>Methods An innovative technique is presented for scaling TQA by creating a new kind of subject-based AbN called a subtaxonomy for large hierarchies. New hypotheses about concentrations of erroneous concepts within the AbN are introduced to guide scalable TQA.<\/jats:p>\n               <jats:p>Results We test the TQA methodology for a subject-based subtaxonomy for the Bleeding subhierarchy in SNOMED's large Clinical finding hierarchy. To test the error concentration hypotheses, three domain experts reviewed a sample of 300 concepts. A consensus-based evaluation identified 87 erroneous concepts. The subtaxonomy-based TQA methodology was shown to uncover statistically significantly more erroneous concepts when compared to a control sample.<\/jats:p>\n               <jats:p>Discussion The scalability of TQA methodologies is a challenge for large standards systems like SNOMED. We demonstrated innovative subject-based TQA techniques by identifying groups of concepts with a higher likelihood of having errors within the subtaxonomy. Scalability is achieved by reviewing a large hierarchy by subject.<\/jats:p>\n               <jats:p>Conclusions An innovative methodology for scaling the derivation of AbNs and a TQA methodology was shown to perform successfully for the largest hierarchy of SNOMED.<\/jats:p>","DOI":"10.1136\/amiajnl-2014-003151","type":"journal-article","created":{"date-parts":[[2014,10,22]],"date-time":"2014-10-22T03:14:01Z","timestamp":1413947641000},"page":"507-518","update-policy":"http:\/\/dx.doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":31,"title":["Scalable quality assurance for large SNOMED CT hierarchies using subject-based subtaxonomies"],"prefix":"10.1093","volume":"22","author":[{"given":"Christopher","family":"Ochs","sequence":"first","affiliation":[{"name":"Computer Science Department, New Jersey Institute of Technology, Newark, New Jersey, USA"}]},{"given":"James","family":"Geller","sequence":"additional","affiliation":[{"name":"Computer Science Department, New Jersey Institute of Technology, Newark, New Jersey, USA"}]},{"given":"Yehoshua","family":"Perl","sequence":"additional","affiliation":[{"name":"Computer Science Department, New Jersey Institute of Technology, Newark, New Jersey, USA"}]},{"given":"Yan","family":"Chen","sequence":"additional","affiliation":[{"name":"Computer Information Systems Department, BMCC, CUNY, New York, New York, USA"}]},{"given":"Junchuan","family":"Xu","sequence":"additional","affiliation":[{"name":"Division of Knowledge Informatics, NYU, New York, New York, USA"}]},{"given":"Hua","family":"Min","sequence":"additional","affiliation":[{"name":"Department of Health Administration and Policy, George Mason University, Fairfax, Virginia, USA"}]},{"given":"James T","family":"Case","sequence":"additional","affiliation":[{"name":"NLM\/NIH, Bethesda, Maryland, USA"}]},{"given":"Zhi","family":"Wei","sequence":"additional","affiliation":[{"name":"Computer Science Department, New Jersey Institute of Technology, Newark, New Jersey, USA"}]}],"member":"286","published-online":{"date-parts":[[2015,10,21]]},"reference":[{"key":"2020110613011569900_R1","first-page":"662","article-title":"SNOMED clinical terms: overview of the development process and project status","author":"Stearns","year":"2001","journal-title":"Proc AMIA Symp"},{"key":"2020110613011569900_R2","first-page":"7","article-title":"SNOMED CT survey: an assessment of implementation in EMR\/EHR applications","volume":"5","author":"Giannangelo","year":"2008","journal-title":"Perspect Health Inf Manag"},{"key":"2020110613011569900_R3","first-page":"815","article-title":"Using SNOMED CT codes for coding information in electronic health records for stroke patients","volume":"124","author":"van der Kooij","year":"2006","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R4","first-page":"361","article-title":"SNOMED CT: electronic health record enhances anesthesia patient safety","volume":"73","author":"Elevitch","year":"2005","journal-title":"AANA J"},{"key":"2020110613011569900_R5","first-page":"59","article-title":"Standard terminology helps advance EHR","volume":"74","author":"Dougherty","year":"2003","journal-title":"J AHIMA"},{"key":"2020110613011569900_R6","first-page":"44589","article-title":"Health information technology: initial set of standards, implementation specifications, and certification criteria for electronic health record technology; interim final rule","volume":"75","author":"Office of the National Coordinator for Health Information Technology, Department of Health and Human Services","year":"2010","journal-title":". Fed Regist"},{"key":"2020110613011569900_R7","first-page":"245","article-title":"eHealth interoperability","volume":"134","author":"Hammond","year":"2008","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R8","first-page":"66","article-title":"SNOMED CT helps drive EHR success","volume":"76","author":"Giannangelo","year":"2005","journal-title":"J AHIMA"},{"key":"2020110613011569900_R9","first-page":"279","article-title":"SNOMED-CT: The advanced terminology and coding system for eHealth","volume":"121","author":"Donnelly","year":"2006","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R10","author":"EHR Incentive Programs"},{"key":"2020110613011569900_R11","author":"Mostashari"},{"key":"2020110613011569900_R12"},{"key":"2020110613011569900_R13","first-page":"75","article-title":"Biomedical ontologies: a functional perspective","volume-title":"Brief Bioinform","author":"Rubin","year":"2008"},{"key":"2020110613011569900_R14","first-page":"67","article-title":"Biomedical ontologies in action: role in knowledge management, data integration and decision support","volume":"1","author":"Bodenreider","year":"2008","journal-title":"Yearb Med Inform"},{"key":"2020110613011569900_R15","first-page":"163\u201382","volume-title":"Medical concept representation","author":"Chute","year":"2005;8:"},{"key":"2020110613011569900_R16","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1197\/jamia.M2036","article-title":"Auditing as part of the terminology design life cycle","volume":"13","author":"Min","year":"2006","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613011569900_R17","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1016\/j.jbi.2006.12.003","article-title":"Structural methodologies for auditing SNOMED","volume":"40","author":"Wang","year":"2007","journal-title":"J Biomed Inform"},{"key":"2020110613011569900_R18","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.jbi.2011.08.013","article-title":"Abstraction of complex concepts with a refined partial-area taxonomy of SNOMED","volume":"45","author":"Wang","year":"2012","journal-title":"J Biomed Inform"},{"key":"2020110613011569900_R19","first-page":"314","article-title":"Analysis of error concentrations in SNOMED","author":"Halper","year":"2007","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R20","first-page":"1071","article-title":"Scalability of abstraction-network-based quality assurance to large SNOMED hierarchies","author":"Ochs","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2011.08.016","article-title":"Auditing complex concepts of SNOMED using a refined hierarchical abstraction network","volume":"45","author":"Wang","year":"2012","journal-title":"J Biomed Inform"},{"issue":"(Suppl 1)","key":"2020110613011569900_R22","doi-asserted-by":"crossref","first-page":"i36","DOI":"10.1136\/amiajnl-2011-000341","article-title":"A survey of SNOMED CT direct users, 2010: impressions and preferences regarding content and quality","volume":"18","author":"Elhanan","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613011569900_R23","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1016\/j.jbi.2009.03.003","article-title":"A review of auditing methods applied to the content of controlled biomedical terminologies","volume":"42","author":"Zhu","year":"2009","journal-title":"J Biomed Inform"},{"key":"2020110613011569900_R24","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1197\/jamia.M2541","article-title":"Auditing the semantic completeness of SNOMED CT using formal concept analysis","volume":"16","author":"Jiang","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613011569900_R25","doi-asserted-by":"crossref","first-page":"432","DOI":"10.1136\/amiajnl-2010-000045","article-title":"Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications","volume":"18","author":"Rector","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613011569900_R26","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.jbi.2011.10.002","article-title":"Lexically suggest, logically define: quality assurance of the use of qualifiers and expected results of post-coordination in SNOMED CT","volume":"45","author":"Rector","year":"2011","journal-title":"J Biomed Inform"},{"key":"2020110613011569900_R27","first-page":"773","article-title":"Semantic clarification of the representation of procedures and diseases in SNOMED((R))CT","volume":"116","author":"Schulz","year":"2005","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R28","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1055\/s-0038-1634088","article-title":"The semantics of procedures and diseases in SNOMED CT","volume":"45","author":"Schulz","year":"2006","journal-title":"Methods Inf Med"},{"issue":"(Suppl 1)","key":"2020110613011569900_R29","doi-asserted-by":"crossref","first-page":"S86","DOI":"10.1016\/j.ijmedinf.2008.06.004","article-title":"Suntisrivaraporn B, Baader F, et al. SNOMED reaching its adolescence: ontologists\u2019 and logicians\u2019 health check","volume":"78","author":"Schulz","year":"2009","journal-title":"Int J Med Inform"},{"issue":"(Pt 1)","key":"2020110613011569900_R30","first-page":"482","article-title":"Ontology-based error detection in SNOMED-CT","volume":"107","author":"Ceusters","year":"2004","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R31","first-page":"1020","article-title":"Crowdsourcing the verification of relationships in biomedical ontologies","author":"Mortensen","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R32","first-page":"453","article-title":"Crowdsourcing user studies with Mechanical Turk","author":"Kittur","year":"2008"},{"key":"2020110613011569900_R33","first-page":"773","article-title":"Identifying problematic concepts in SNOMED CT using a lexical approach","volume":"192","author":"Agrawal","year":"2013","journal-title":"Stud Health Technol Inform"},{"key":"2020110613011569900_R34","first-page":"17","article-title":"Identifying inconsistencies in SNOMED CT problem lists using structural indicators","author":"Agrawal","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R35","first-page":"212","article-title":"Dissimilarities in the logical modeling of apparently similar concepts in SNOMED CT","volume":"2010","author":"Agrawal","year":"2010","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R36","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/j.artmed.2006.12.003","article-title":"Investigating subsumption in SNOMED CT: an exploration into large description logic-based biomedical terminologies","volume":"39","author":"Bodenreider","year":"2007","journal-title":"Artif Intell Med"},{"key":"2020110613011569900_R37","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/S1386-5056(02)00051-5","article-title":"Assessing the consistency of a biomedical terminology through lexical knowledge","volume":"67","author":"Bodenreider","year":"2002","journal-title":"Int J Med Inform"},{"key":"2020110613011569900_R38","doi-asserted-by":"crossref","first-page":"648","DOI":"10.1002\/cfg.445","article-title":"Overview and utilization of the NCI thesaurus","volume":"5","author":"Fragoso","year":"2004","journal-title":"Comp Funct Genomics"},{"key":"2020110613011569900_R39","first-page":"237","article-title":"New abstraction networks and a new visualization tool in support of auditing the SNOMED CT Content","author":"Geller","year":"2012","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613011569900_R40","doi-asserted-by":"crossref","first-page":"1042","DOI":"10.1016\/j.jbi.2012.05.006","article-title":"A study of terminology auditors\u2019 performance for UMLS semantic type assignments","volume":"45","author":"Gu","year":"2012","journal-title":"J Biomed Inform"},{"key":"2020110613011569900_R41"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/3\/507\/34145482\/amiajnl-2014-003151.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/3\/507\/34145482\/amiajnl-2014-003151.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:48:01Z","timestamp":1604688481000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/22\/3\/507\/772028"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,5,1]]},"references-count":41,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2015,10,21]]},"published-print":{"date-parts":[[2015,5,1]]}},"URL":"https:\/\/doi.org\/10.1136\/amiajnl-2014-003151","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,5]]},"published":{"date-parts":[[2015,5,1]]}}}