{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T08:22:16Z","timestamp":1773217336890,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":35,"publisher":"ACM","license":[{"start":{"date-parts":[[2013,5,2]],"date-time":"2013-05-02T00:00:00Z","timestamp":1367452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2013,5,2]]},"DOI":"10.1145\/2464464.2464482","type":"proceedings-article","created":{"date-parts":[[2013,7,16]],"date-time":"2013-07-16T18:06:58Z","timestamp":1373998018000},"page":"262-271","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Mechanical turk as an ontology engineer?"],"prefix":"10.1145","author":[{"given":"Natalya F.","family":"Noy","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA"}]},{"given":"Jonathan","family":"Mortensen","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}]},{"given":"Mark A.","family":"Musen","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}]},{"given":"Paul R.","family":"Alexander","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}]}],"member":"320","published-online":{"date-parts":[[2013,5,2]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CTS.2011.5928720"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/11926078_53"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2008.09.001"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02121-3_65"},{"key":"e_1_3_2_1_8_1","first-page":"8","article-title":"Creating the Gene Ontology resource: design and implementation","volume":"11","author":"GOConsortium","year":"2001","unstructured":"GOConsortium . Creating the Gene Ontology resource: design and implementation . Genome Res 11 , 8 ( 2001 ), 1425--33. GOConsortium. Creating the Gene Ontology resource: design and implementation. Genome Res 11, 8 (2001), 1425--33.","journal-title":"Genome Res"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4379(00)00010-7"},{"key":"e_1_3_2_1_10_1","volume-title":"Carothe common anatomy reference ontology. Anatomy Ontologies for Bioinformatics","author":"Haendel M.","year":"2008","unstructured":"Haendel , M. , Neuhaus , F. , Osumi-Sutherland , D. , Mabee , P. , Mejino , J. , Mungall , C. , and Smith , B . Carothe common anatomy reference ontology. Anatomy Ontologies for Bioinformatics ( 2008 ), 327--349. Haendel, M., Neuhaus, F., Osumi-Sutherland, D., Mabee, P., Mejino, J., Mungall, C., and Smith, B. Carothe common anatomy reference ontology. Anatomy Ontologies for Bioinformatics (2008), 327--349."},{"key":"e_1_3_2_1_11_1","volume-title":"WWW 2009 Workshop: Linked Data on the Web","author":"Hausenblas M.","year":"2009","unstructured":"Hausenblas , M. , Troncy , R. , Raimond , Y. , and Brger , T . Interlinking multimedia: How to apply linked data principles to multimedia fragments . In WWW 2009 Workshop: Linked Data on the Web ( 2009 ). Hausenblas, M., Troncy, R., Raimond, Y., and Brger, T. Interlinking multimedia: How to apply linked data principles to multimedia fragments. In WWW 2009 Workshop: Linked Data on the Web (2009)."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_13_1","volume-title":"Twenty-Sixth AAAI Conference on Artificial Intelligence","author":"Lin C. H.","year":"2012","unstructured":"Lin , C. H. , Mausam , and Weld, D. S . Dynamically switching between synergistic workows for crowdsourcing . In Twenty-Sixth AAAI Conference on Artificial Intelligence ( 2012 ). Lin, C. H., Mausam, and Weld, D. S. Dynamically switching between synergistic workows for crowdsourcing. In Twenty-Sixth AAAI Conference on Artificial Intelligence (2012)."},{"key":"e_1_3_2_1_14_1","volume-title":"Proceedings of the Workshop on Knowledge Injection into and Extraction from Linked Data at EKAW","author":"Markotschi T.","year":"2010","unstructured":"Markotschi , T. , and V\u00f6lker , J . GuessWhat?! - Human Intelligence for Mining Linked Data . In Proceedings of the Workshop on Knowledge Injection into and Extraction from Linked Data at EKAW ( 2010 ). Markotschi, T., and V\u00f6lker, J. GuessWhat?! - Human Intelligence for Mining Linked Data. In Proceedings of the Workshop on Knowledge Injection into and Extraction from Linked Data at EKAW (2010)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2008.4497419"},{"key":"e_1_3_2_1_17_1","volume-title":"Workshop on Social Computing and User Generated Content in conjunction with ACM Conference on Electronic Commerce (ACM-EC 2012)","author":"Minder P.","year":"2012","unstructured":"Minder , P. , Seuken , S. , Bernstein , A. , and Zollinger , M . Crowdmanager-combinatorial allocation and pricing of crowdsourcing tasks with time constraints . In Workshop on Social Computing and User Generated Content in conjunction with ACM Conference on Electronic Commerce (ACM-EC 2012) (Valencia, Spain , 2012 ), 1--18. Minder, P., Seuken, S., Bernstein, A., and Zollinger, M. Crowdmanager-combinatorial allocation and pricing of crowdsourcing tasks with time constraints. In Workshop on Social Computing and User Generated Content in conjunction with ACM Conference on Electronic Commerce (ACM-EC 2012) (Valencia, Spain, 2012), 1--18."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000523"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/505168.505170"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88564-1_24"},{"key":"e_1_3_2_1_21_1","volume-title":"Ontology engineering through microtask crowdsourcing. Under review","author":"Noy N. F.","year":"2013","unstructured":"Noy , N. F. , Mortensen , J. , Alexander , P. R. , and Musen , M. A . Ontology engineering through microtask crowdsourcing. Under review ( 2013 ). Noy, N. F., Mortensen, J., Alexander, P. R., and Musen, M. A. Ontology engineering through microtask crowdsourcing. Under review (2013)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_3_2_1_23_1","volume-title":"Galaxy zoo: exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925","author":"Raddick M.","year":"2009","unstructured":"Raddick , M. , Bracey , G. , Gay , P. , Lintott , C. , Murray , P. , Schawinski , K. , Szalay , A. , and Vandenberg , J . Galaxy zoo: exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925 ( 2009 ). Raddick, M., Bracey, G., Gay, P., Lintott, C., Murray, P., Schawinski, K., Szalay, A., and Vandenberg, J. Galaxy zoo: exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925 (2009)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_33"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.54.2.93"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-87696-0_28"},{"key":"e_1_3_2_1_27_1","volume-title":"2nd workshop on consuming Linked Data COLD2011 co-located with the 10th International Semantic Web Conference ISWC 2011","author":"Simperl E.","year":"2011","unstructured":"Simperl , E. , Norton , B. , and Vrandecic , D . Crowdsourcing tasks in linked data management . In 2nd workshop on consuming Linked Data COLD2011 co-located with the 10th International Semantic Web Conference ISWC 2011 ( Bonn, Germany , 2011 ). Simperl, E., Norton, B., and Vrandecic, D. Crowdsourcing tasks in linked data management. In 2nd workshop on consuming Linked Data COLD2011 co-located with the 10th International Semantic Web Conference ISWC 2011 (Bonn, Germany, 2011)."},{"key":"e_1_3_2_1_28_1","volume-title":"Questions about Questions: Inquiries Into the Coqnitive Bases of Surveys","author":"Tanur J. M.","year":"1992","unstructured":"Tanur , J. M. Questions about Questions: Inquiries Into the Coqnitive Bases of Surveys . Russell Sage Foundation Publications , 1992 . Tanur, J. M. Questions about Questions: Inquiries Into the Coqnitive Bases of Surveys. Russell Sage Foundation Publications, 1992."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1982185.1982542"},{"key":"e_1_3_2_1_30_1","first-page":"11","article-title":"Webprot\u00e9g\u00e9: A distributed ontology editor and knowledge acquisition tool for the web","author":"Tudorache T.","year":"2011","unstructured":"Tudorache , T. , Nyulas , C. , Noy , N. F. , and Musen , M. A . Webprot\u00e9g\u00e9: A distributed ontology editor and knowledge acquisition tool for the web . Semantic Web Journal 11 - 165 ( 2011 ). Tudorache, T., Nyulas, C., Noy, N. F., and Musen, M. A. Webprot\u00e9g\u00e9: A distributed ontology editor and knowledge acquisition tool for the web. Semantic Web Journal 11-165 (2011).","journal-title":"Semantic Web Journal"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_3_2_1_32_1","volume-title":"Thirty Third International Conference on Information Systems (ICIS)","author":"Wang J.","year":"2012","unstructured":"Wang , J. , Ghose , A. , and Ipeirotis , P . Bonus, disclosure, and choice: What motivates the creation of high-quality paid reviews? In Thirty Third International Conference on Information Systems (ICIS) ( Orlando, FL , 2012 ). Wang, J., Ghose, A., and Ipeirotis, P. Bonus, disclosure, and choice: What motivates the creation of high-quality paid reviews? In Thirty Third International Conference on Information Systems (ICIS) (Orlando, FL, 2012)."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441846"},{"key":"e_1_3_2_1_34_1","volume-title":"Web Server issue","author":"Whetzel P. L.","year":"2011","unstructured":"Whetzel , P. L. , Noy , N. F. , Shah , N. H. , Alexander , P. R. , Nyulas , C. I. , Tudorache , T. , and Musen , M. A . Bioportal: Enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications. Nucleic Acids Research (NAR) 39 , Web Server issue ( 2011 ), W541--5. Whetzel, P. L., Noy, N. F., Shah, N. H., Alexander, P. R., Nyulas, C. I., Tudorache, T., and Musen, M. A. Bioportal: Enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications. Nucleic Acids Research (NAR) 39, Web Server issue (2011), W541--5."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/11762256_6"}],"event":{"name":"WebSci '13: Web Science 2013","location":"Paris France","acronym":"WebSci '13","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["Proceedings of the 5th Annual ACM Web Science Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2464464.2464482","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2464464.2464482","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:39:04Z","timestamp":1750235944000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2464464.2464482"}},"subtitle":["using microtasks as a component of an ontology-engineering workflow"],"short-title":[],"issued":{"date-parts":[[2013,5,2]]},"references-count":35,"alternative-id":["10.1145\/2464464.2464482","10.1145\/2464464"],"URL":"https:\/\/doi.org\/10.1145\/2464464.2464482","relation":{},"subject":[],"published":{"date-parts":[[2013,5,2]]},"assertion":[{"value":"2013-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}