{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:24:50Z","timestamp":1750307090493,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":27,"publisher":"ACM","license":[{"start":{"date-parts":[[2012,10,29]],"date-time":"2012-10-29T00:00:00Z","timestamp":1351468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2012,10,29]]},"DOI":"10.1145\/2390803.2390816","type":"proceedings-article","created":{"date-parts":[[2012,11,13]],"date-time":"2012-11-13T15:04:07Z","timestamp":1352819047000},"page":"29-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Crowdsourcing micro-level multimedia annotations"],"prefix":"10.1145","author":[{"given":"Sunghyun","family":"Park","sequence":"first","affiliation":[{"name":"University of Southern California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gelareh","family":"Mohammadi","sequence":"additional","affiliation":[{"name":"\u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne (EPFL), Lausanne, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ron","family":"Artstein","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Louis-Philippe","family":"Morency","sequence":"additional","affiliation":[{"name":"University of Southern California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2012,10,29]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Allwood J. Cerrato L. Dybkjaer L. Jokinen K. Navarretta C. and Paggio P. 2004. The MUMIN multimodal coding scheme. NorFA yearbook 2005 129--157.  Allwood J. Cerrato L. Dybkjaer L. Jokinen K. Navarretta C. and Paggio P. 2004. The MUMIN multimodal coding scheme. NorFA yearbook 2005 129--157."},{"volume-title":"Proc. of the 6th Int'l AAAI Conf. on Weblogs and Social Media. ICWSM '12","author":"Biel J. I.","key":"e_1_3_2_1_2_1","unstructured":"Biel , J. I. and Gatica-Perez , D . 2012. The good, the bad, and the angry: Analyzing crowdsourced impressions of vloggers . In Proc. of the 6th Int'l AAAI Conf. on Weblogs and Social Media. ICWSM '12 . Biel, J. I. and Gatica-Perez, D. 2012. The good, the bad, and the angry: Analyzing crowdsourced impressions of vloggers. In Proc. of the 6th Int'l AAAI Conf. on Weblogs and Social Media. ICWSM '12."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Dasiopoulou S. Giannakidou E. Litos G. Malasioti P. and Kompatsiaris Y. 2011. A survey of semantic image and video annotation tools. In Knowledge-Driven Multimedia Information Extraction and Ontology Evolution G. Paliouras C. Spyropoulos and G. Tsatsaronis Ed. Lecture Notes in Computer Science. Springer Berlin 196--239.   Dasiopoulou S. Giannakidou E. Litos G. Malasioti P. and Kompatsiaris Y. 2011. A survey of semantic image and video annotation tools. In Knowledge-Driven Multimedia Information Extraction and Ontology Evolution G. Paliouras C. Spyropoulos and G. Tsatsaronis Ed. Lecture Notes in Computer Science. Springer Berlin 196--239.","DOI":"10.1007\/978-3-642-20795-2_8"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753688"},{"volume-title":"Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 30--34","author":"Gao Q.","key":"e_1_3_2_1_5_1","unstructured":"Gao , Q. and Vogel , S . 2010. Consensus versus expertise: A case study of word alignment with Mechanical Turk . In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 30--34 . Gao, Q. and Vogel, S. 2010. Consensus versus expertise: A case study of word alignment with Mechanical Turk. In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 30--34."},{"volume-title":"Proc. of the NAACL HLT 2009 Workshop on Active Learning for Natural Lang. Processing, 27--35","author":"Hsueh P. Y.","key":"e_1_3_2_1_6_1","unstructured":"Hsueh , P. Y. , Melville , P. , and Sindhwani , V . 2009. Data quality from crowdsourcing: a study of annotation selection criteria . In Proc. of the NAACL HLT 2009 Workshop on Active Learning for Natural Lang. Processing, 27--35 . Hsueh, P. Y., Melville, P., and Sindhwani, V. 2009. Data quality from crowdsourcing: a study of annotation selection criteria. In Proc. of the NAACL HLT 2009 Workshop on Active Learning for Natural Lang. Processing, 27--35."},{"key":"e_1_3_2_1_7_1","volume-title":"Proc. of the 11th Int'l Conf. on Autonomous Agents and Multiagent Systems. AAMAS '12, 63--70","author":"Kang S.","year":"2012","unstructured":"Kang , S. , Gratch , J. , Sidner , C. , Artstein , R. , Huang , L. , Morency , L. 2012 . Towards building a virtual counselor: Modeling nonverbal behavior during intimate self-disclosure . In Proc. of the 11th Int'l Conf. on Autonomous Agents and Multiagent Systems. AAMAS '12, 63--70 . Kang, S., Gratch, J., Sidner, C., Artstein, R., Huang, L., Morency, L. 2012. Towards building a virtual counselor: Modeling nonverbal behavior during intimate self-disclosure. In Proc. of the 11th Int'l Conf. on Autonomous Agents and Multiagent Systems. AAMAS '12, 63--70."},{"key":"e_1_3_2_1_8_1","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff K.","year":"1980","unstructured":"Krippendorff , K. 1980 . Content Analysis: An Introduction to Its Methodology . Sage , Beverly Hills, CA . Krippendorff, K. 1980. Content Analysis: An Introduction to Its Methodology. Sage, Beverly Hills, CA."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.2307\/271061"},{"volume-title":"Proc. of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Eval. CSE '10, 21--26","author":"Le J.","key":"e_1_3_2_1_10_1","unstructured":"Le , J. , Edmonds , A. , Hester , V. , and Biewald , L . 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In Proc. of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Eval. CSE '10, 21--26 . Le, J., Edmonds, A., Hester, V., and Biewald, L. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Eval. CSE '10, 21--26."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1126004.1126005"},{"volume-title":"Proc. of the 2010 IEEE Int'l Conf. on Acoustics Speech and Sig. Process. ICASSP '10, 5270--5273","author":"Marge M.","key":"e_1_3_2_1_12_1","unstructured":"Marge , M. , Banerjee , S. , and Rudnicky , A. I . 2010. Using the Amazon Mechanical Turk for transcription of spoken language . In Proc. of the 2010 IEEE Int'l Conf. on Acoustics Speech and Sig. Process. ICASSP '10, 5270--5273 . Marge, M., Banerjee, S., and Rudnicky, A. I. 2010. Using the Amazon Mechanical Turk for transcription of spoken language. In Proc. of the 2010 IEEE Int'l Conf. on Acoustics Speech and Sig. Process. ICASSP '10, 5270--5273."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_3_2_1_14_1","first-page":"207","article-title":"Cheap, fast and good enough: automatic speech recognition with non-expert transcription. In Proc. of the 2010 NAACL Human","volume":"10","author":"Novotney S.","year":"2010","unstructured":"Novotney , S. and Callison-Burch , C. 2010 . Cheap, fast and good enough: automatic speech recognition with non-expert transcription. In Proc. of the 2010 NAACL Human Lang. Tech. HLT ' 10 , 207 -- 215 . Novotney, S. and Callison-Burch, C. 2010. Cheap, fast and good enough: automatic speech recognition with non-expert transcription. In Proc. of the 2010 NAACL Human Lang. Tech. HLT '10, 207--215.","journal-title":"Lang. Tech. HLT '"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1180995.1181044"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2009.11.014"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"volume-title":"Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 139--147","author":"Rashtchian C.","key":"e_1_3_2_1_19_1","unstructured":"Rashtchian , C. , Young , P. , Hodosh , M. , and Hockenmaier , J . 2010. Collecting image annotations using Amazon's Mechanical Turk . In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 139--147 . Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J. 2010. Collecting image annotations using Amazon's Mechanical Turk. In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Lang. Data with Amazon's Mech. Turk. CSLDAMT '10, 139--147."},{"volume-title":"Proc. of the 4th Int'l Conf. on Affective Computing and Intelligent Interaction. ACII '11","author":"Riek L.","key":"e_1_3_2_1_20_1","unstructured":"Riek , L. , O'Connor , M. , and Robinson , P . 2011. Guess what? A game for affective annotation of video using crowd sourcing . In Proc. of the 4th Int'l Conf. on Affective Computing and Intelligent Interaction. ACII '11 , 277--285. Riek, L., O'Connor, M., and Robinson, P. 2011. Guess what? A game for affective annotation of video using crowd sourcing. In Proc. of the 4th Int'l Conf. on Affective Computing and Intelligent Interaction. ACII '11, 277--285."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"volume-title":"Proc. of the Conf. on Empirical Methods in Natural Lang. Proc. EMNLP '08","author":"Snow R.","key":"e_1_3_2_1_23_1","unstructured":"Snow , R. , O'Connor , B. , Jurafsky , D. and Ng , A. Y . 2008. Cheap and fast but is it good?: Evaluating non-expert annotations for natural language tasks . In Proc. of the Conf. on Empirical Methods in Natural Lang. Proc. EMNLP '08 , 254--263. Snow, R., O'Connor, B., Jurafsky, D. and Ng, A. Y. 2008. Cheap and fast but is it good?: Evaluating non-expert annotations for natural language tasks. In Proc. of the Conf. on Empirical Methods in Natural Lang. Proc. EMNLP '08, 254--263."},{"volume-title":"Proc. of the 2010 IEEE Computer Vision and Pattern Recog. Workshops. CVPRW '10, 17--24","author":"Spiro I.","key":"e_1_3_2_1_24_1","unstructured":"Spiro , I. , Taylor , G. , Williams , G. , and Bregler , C . 2010. Hands by hand: Crowd-sourced motion tracking for gesture annotation . In Proc. of the 2010 IEEE Computer Vision and Pattern Recog. Workshops. CVPRW '10, 17--24 . Spiro, I., Taylor, G., Williams, G., and Bregler, C. 2010. Hands by hand: Crowd-sourced motion tracking for gesture annotation. In Proc. of the 2010 IEEE Computer Vision and Pattern Recog. Workshops. CVPRW '10, 17--24."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Vondrick C. Ramanan D. and Patterson D. 2010. Efficiently scaling up video annotation with crowdsourced marketplaces. Computer Vision - ECCV 2010 6314 610--623.   Vondrick C. Ramanan D. and Patterson D. 2010. Efficiently scaling up video annotation with crowdsourced marketplaces. Computer Vision - ECCV 2010 6314 610--623.","DOI":"10.1007\/978-3-642-15561-1_44"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1979742.1979803"},{"volume-title":"Proc. of the IEEE 3rd Int'l Conf. on Social Computing. SocialComp '11","author":"Yuen M. C.","key":"e_1_3_2_1_27_1","unstructured":"Yuen , M. C. , King , I. , and Leung , K. S . 2011. A survey of crowdsourcing systems . In Proc. of the IEEE 3rd Int'l Conf. on Social Computing. SocialComp '11 , 766--773. Yuen, M. C., King, I., and Leung, K. S. 2011. A survey of crowdsourcing systems. In Proc. of the IEEE 3rd Int'l Conf. on Social Computing. SocialComp '11, 766--773."}],"event":{"name":"MM '12: ACM Multimedia Conference","sponsor":["SIGMM ACM Special Interest Group on Multimedia"],"location":"Nara Japan","acronym":"MM '12"},"container-title":["Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2390803.2390816","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2390803.2390816","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:34:30Z","timestamp":1750239270000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2390803.2390816"}},"subtitle":["the challenges of evaluation and interface"],"short-title":[],"issued":{"date-parts":[[2012,10,29]]},"references-count":27,"alternative-id":["10.1145\/2390803.2390816","10.1145\/2390803"],"URL":"https:\/\/doi.org\/10.1145\/2390803.2390816","relation":{},"subject":[],"published":{"date-parts":[[2012,10,29]]},"assertion":[{"value":"2012-10-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}