{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T10:56:53Z","timestamp":1772794613709,"version":"3.50.1"},"reference-count":27,"publisher":"Oxford University Press (OUP)","issue":"3","content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2015,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Objectives The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification.<\/jats:p>\n               <jats:p>Methods We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships.<\/jats:p>\n               <jats:p>Results The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (\u03ba) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, \u2018septic shock is a soft-tissue infection\u2019).<\/jats:p>\n               <jats:p>Discussion The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks.<\/jats:p>\n               <jats:p>Conclusions We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.<\/jats:p>","DOI":"10.1136\/amiajnl-2014-002901","type":"journal-article","created":{"date-parts":[[2014,10,24]],"date-time":"2014-10-24T01:53:02Z","timestamp":1414115582000},"page":"640-648","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":40,"title":["Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT"],"prefix":"10.1093","volume":"22","author":[{"given":"Jonathan M","family":"Mortensen","sequence":"first","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"},{"name":"Biomedical Informatics Training Program, Stanford University, Stanford, California, USA"}]},{"given":"Evan P","family":"Minty","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Training Program, Stanford University, Stanford, California, USA"},{"name":"Faculty of Medicine, University of Calgary, Calgary, Canada"}]},{"given":"Michael","family":"Januszyk","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Training Program, Stanford University, Stanford, California, USA"}]},{"given":"Timothy E","family":"Sweeney","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}]},{"given":"Alan L","family":"Rector","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Manchester, Manchester, UK"}]},{"given":"Natalya F","family":"Noy","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"},{"name":"Google Inc., Mountain View, California, USA"}]},{"given":"Mark A","family":"Musen","sequence":"additional","affiliation":[{"name":"Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"},{"name":"Biomedical Informatics Training Program, Stanford University, Stanford, California, USA"}]}],"member":"286","published-online":{"date-parts":[[2014,10,23]]},"reference":[{"key":"2020110612593229500_R1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe","year":"2006","journal-title":"Wired Mag"},{"key":"2020110612593229500_R2","doi-asserted-by":"crossref","first-page":"1403","DOI":"10.1145\/1978942.1979148","article-title":"Human computation: a survey and taxonomy of a growing field","volume-title":"Proceedings of the 2011 annual conference on Human factors in computing systems\u2014CHI'11","author":"Quinn","year":"2011"},{"key":"2020110612593229500_R3","doi-asserted-by":"crossref","first-page":"1179","DOI":"10.1111\/j.1365-2966.2008.13689.x","article-title":"Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey","volume":"389","author":"Lintott","year":"2008","journal-title":"Mon Not R Astron Soc"},{"key":"2020110612593229500_R4","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1038\/nature09304","article-title":"Predicting protein structures with a multiplayer online game","volume":"466","author":"Cooper","year":"2010","journal-title":"Nature"},{"key":"2020110612593229500_R5","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-540-92673-3","volume-title":"Handbook on ontologies","author":"Staab","year":"2009","edition":"2nd edn"},{"key":"2020110612593229500_R6","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1093\/bib\/bbl027","article-title":"Bio-ontologies: current trends and future directions","volume":"7","author":"Bodenreider","year":"2006","journal-title":"Brief Bioinform"},{"key":"2020110612593229500_R7","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1093\/bib\/bbm059","article-title":"Biomedical ontologies: a functional perspective","volume":"9","author":"Rubin","year":"2008","journal-title":"Brief Bioinform"},{"key":"2020110612593229500_R8","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/1471-2105-9-78","article-title":"OpenDMAP: an open source, ontology-driven concept analysis engine, with applications to capturing knowledge regarding protein transport, protein interactions and cell-type-specific gene expression","volume":"9","author":"Hunter","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2020110612593229500_R9","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1186\/1752-0509-5-124","article-title":"Integrating systems biology models and biomedical ontologies","volume":"5","author":"Hoehndorf","year":"2011","journal-title":"BMC Syst Biol"},{"key":"2020110612593229500_R10","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1038\/ng1165","article-title":"Module networks: identifying regulatory modules and their condition-specific regulators from gene expression data","volume":"34","author":"Segal","year":"2003","journal-title":"Nat Genet"},{"key":"2020110612593229500_R11","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1038\/clpt.2013.47","article-title":"Pharmacovigilance using clinical notes","volume":"93","author":"LePendu","year":"2013","journal-title":"Clin Pharmacol Ther"},{"key":"2020110612593229500_R12","doi-asserted-by":"crossref","first-page":"W541","DOI":"10.1093\/nar\/gkr469","article-title":"BioPortal: enhanced functionality via new web services from the National Center for Biomedical Ontology to access and use ontologies in software applications","volume":"39","author":"Whetzel","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2020110612593229500_R13","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1056\/NEJMp1006114","article-title":"The \u201cmeaningful use\u201d regulation for electronic health records","volume":"363","author":"Blumenthal","year":"2010","journal-title":"N Engl J Med"},{"key":"2020110612593229500_R14","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1055\/s-0038-1634000","article-title":"A terminological and ontological analysis of the NCI Thesaurus","volume":"44","author":"Ceusters","year":"2005","journal-title":"Methods Inf Med"},{"key":"2020110612593229500_R15","doi-asserted-by":"crossref","first-page":"432","DOI":"10.1136\/amiajnl-2010-000045","article-title":"Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications","volume":"18","author":"Rector","year":"2011","journal-title":"J Am Med Informatics Assoc"},{"key":"2020110612593229500_R16","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1016\/j.jbi.2009.03.003","article-title":"A review of auditing methods applied to the content of controlled biomedical terminologies","volume":"42","author":"Zhu","year":"2009","journal-title":"J Biomed Inform"},{"key":"2020110612593229500_R17","first-page":"1071","article-title":"Scalability of abstraction-network-based quality assurance to large SNOMED hierarchies","volume":"2013","author":"Ochs","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110612593229500_R18","article-title":"Mechanical Turk as an ontology engineer? Using microtasks as a component of an ontology engineering workflow","author":"Noy","year":"2013"},{"key":"2020110612593229500_R19","first-page":"45","article-title":"Fast classification in Prot\u00e9g\u00e9: Snorocket as an OWL 2 EL reasoner","author":"Lawley","year":"2010"},{"key":"2020110612593229500_R20","first-page":"11","article-title":"The OWL API: a Java API for working with OWL 2 ontologies","author":"Horridge","year":"2009"},{"key":"2020110612593229500_R21","article-title":"Crowdsourcing ontology verification. International Conference on Biomedical Ontologies","author":"Mortensen","year":"2013"},{"key":"2020110612593229500_R22","article-title":"Dynamic Bayesian combination of multiple imperfect classifiers","author":"Simpson","year":"2012"},{"key":"2020110612593229500_R23","volume-title":"The Delphi method: techniques and applications","author":"Linstone","year":"1975"},{"key":"2020110612593229500_R24","article-title":"Free-marginal multirater kappa (multirater \u03bafree): an alternative to Fleiss\u2019 fixed-marginal multirater kappa","author":"Randolph","year":"2005"},{"key":"2020110612593229500_R25","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611970319","volume-title":"The Jackknife, the Bootstrap and other resampling plans","author":"Efron","year":"1982"},{"key":"2020110612593229500_R26","doi-asserted-by":"crossref","first-page":"3587","DOI":"10.1093\/bioinformatics\/bti565","article-title":"Ontological analysis of gene expression data: current tools, limitations, and open problems","volume":"21","author":"Khatri","year":"2005","journal-title":"Bioinformatics"},{"key":"2020110612593229500_R27","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1136\/jamia.2009.000893","article-title":"Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)","volume":"17","author":"Murphy","year":"2010","journal-title":"J Am Med Informatics Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/3\/640\/34145617\/amiajnl-2014-002901.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/3\/640\/34145617\/amiajnl-2014-002901.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:37:53Z","timestamp":1604687873000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/22\/3\/640\/771693"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,10,23]]},"references-count":27,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2014,10,23]]},"published-print":{"date-parts":[[2015,5,1]]}},"URL":"https:\/\/doi.org\/10.1136\/amiajnl-2014-002901","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,5]]},"published":{"date-parts":[[2014,10,23]]}}}