{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T17:04:10Z","timestamp":1770915850573,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,2]],"date-time":"2019-05-02T00:00:00Z","timestamp":1556755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1544753"],"award-info":[{"award-number":["1544753"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,2]]},"DOI":"10.1145\/3290605.3300522","type":"proceedings-article","created":{"date-parts":[[2019,4,29]],"date-time":"2019-04-29T17:36:58Z","timestamp":1556559418000},"page":"1-11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["Crowdsourcing Multi-label Audio Annotation Tasks with Citizen Scientists"],"prefix":"10.1145","author":[{"given":"Mark","family":"Cartwright","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Graham","family":"Dove","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Ana Elisa","family":"M\u00e9ndez M\u00e9ndez","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Juan P.","family":"Bello","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Oded","family":"Nov","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,5,2]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"{n. d.}. Zooniverse. https:\/\/www.zooniverse.org\/. Accessed: 2018-0919.  {n. d.}. Zooniverse. https:\/\/www.zooniverse.org\/. Accessed: 2018-0919."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1121\/1.4709112"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1121\/1.3493436"},{"key":"e_1_3_2_2_4_1","volume-title":"Auditory and non-auditory efects of noise on health. The Lancet 383, 9925","author":"Basner Mathias","year":"2014","unstructured":"Mathias Basner , Wolfgang Babisch , Adrian Davis , Mark Brink , Charlotte Clark , Sabine Janssen , and Stephen Stansfeld . 2014. Auditory and non-auditory efects of noise on health. The Lancet 383, 9925 ( 2014 ), 1325--1332. Mathias Basner, Wolfgang Babisch, Adrian Davis, Mark Brink, Charlotte Clark, Sabine Janssen, and Stephen Stansfeld. 2014. Auditory and non-auditory efects of noise on health. The Lancet 383, 9925 (2014), 1325--1332."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3224204"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2583008.2583011"},{"key":"e_1_3_2_2_7_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.","author":"Bragg Jonathan","year":"2013","unstructured":"Jonathan Bragg and Daniel S Weld . 2013 . Crowdsourcing multi-label classifcation for taxonomy creation . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Jonathan Bragg and Daniel S Weld. 2013. Crowdsourcing multi-label classifcation for taxonomy creation. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/EUSIPCO.2015.7362845"},{"key":"e_1_3_2_2_9_1","volume-title":"Advances in Computers.","author":"Carroll John M","unstructured":"John M Carroll . 1989. Evaluation , description and invention: Paradigms for human-computer interaction . In Advances in Computers. Vol. 29 . Elsevier , 47--77. John M Carroll. 1989. Evaluation, description and invention: Paradigms for human-computer interaction. In Advances in Computers. Vol. 29. Elsevier, 47--77."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134664"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jebo.2013.03.003"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1873951.1873959"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0106508"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557011"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819939"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.biocon.2016.08.020"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557262"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7952261"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1289\/ehp.1307272"},{"key":"e_1_3_2_2_22_1","volume-title":"Proceedings of the International Society for Music Information Retrieval Conference.","author":"Humphrey Eric","year":"2018","unstructured":"Eric Humphrey , Simon Durand , and Brian McFee . 2018 . OpenMIC-2018: an open dataset for multiple instrument recognition . In Proceedings of the International Society for Music Information Retrieval Conference. Eric Humphrey, Simon Durand, and Brian McFee. 2018. OpenMIC-2018: an open dataset for multiple instrument recognition. In Proceedings of the International Society for Music Information Retrieval Conference."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0104_2"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468553"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2957276.2957284"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2015.196"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7952263"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.22323\/2.15030205"},{"key":"e_1_3_2_2_29_1","volume-title":"The epistemic culture in an online citizen science project: Programs, antiprograms and epistemic subjects. Social Studies of Science","author":"Kasperowski Dick","year":"2018","unstructured":"Dick Kasperowski and Thomas Hillman . 2018. The epistemic culture in an online citizen science project: Programs, antiprograms and epistemic subjects. Social Studies of Science ( 2018 ). Dick Kasperowski and Thomas Hillman. 2018. The epistemic culture in an online citizen science project: Programs, antiprograms and epistemic subjects. Social Studies of Science (2018)."},{"key":"e_1_3_2_2_30_1","volume-title":"Proceedings of Americas Conference on Information Systems. 1--11","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk . In Proceedings of Americas Conference on Information Systems. 1--11 . Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk. In Proceedings of Americas Conference on Information Systems. 1--11."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858115"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2966.2008.13689.x"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005995"},{"key":"e_1_3_2_2_36_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.","author":"Mao Andrew","year":"2013","unstructured":"Andrew Mao , Ece Kamar , Yiling Chen , Eric Horvitz , Megan E Schwamb , Chris J Lintott , and Arfon M Smith . 2013 . Volunteering versus work for pay: Incentives and tradeofs in crowdsourcing . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan E Schwamb, Chris J Lintott, and Arfon M Smith. 2013. Volunteering versus work for pay: Incentives and tradeofs in crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1890\/110294"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1940761.1940771"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0090375"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScienceW.2011.14"},{"key":"e_1_3_2_2_42_1","volume-title":"Galaxy Zoo: Motivations of citizen scientists.","author":"Raddick M Jordan","year":"2013","unstructured":"M Jordan Raddick , Georgia Bracey , Pamela L Gay , Chris J Lintott , Carie Cardamone , Phil Murray , Kevin Schawinski , Alexander S Szalay , and Jan Vandenberg . 2013 . Galaxy Zoo: Motivations of citizen scientists. (2013). arXiv:physics.ed-ph\/1303.6886 M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Carie Cardamone, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg. 2013. Galaxy Zoo: Motivations of citizen scientists. (2013). arXiv:physics.ed-ph\/1303.6886"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.3847\/AER2009036"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2013.85"},{"key":"e_1_3_2_2_45_1","volume-title":"Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public understanding of science 23, 1","author":"Riesch Hauke","year":"2014","unstructured":"Hauke Riesch and Clive Potter . 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public understanding of science 23, 1 ( 2014 ), 107--120. Hauke Riesch and Clive Potter. 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public understanding of science 23, 1 (2014), 107--120."},{"key":"e_1_3_2_2_46_1","volume-title":"Proceedigns of the International AAAI Conference on Web and Social Media, 17--21","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets . Proceedigns of the International AAAI Conference on Web and Social Media, 17--21 . Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. Proceedigns of the International AAAI Conference on Web and Social Media, 17--21."},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145238"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1121\/1.4861348"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1080\/01449298208914450"},{"key":"e_1_3_2_2_50_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.","author":"Sigurdsson Gunnar A","year":"2016","unstructured":"Gunnar A Sigurdsson , Olga Russakovsky , Ali Farhadi , Ivan Laptev , and Abhinav Gupta . 2016 . Much ado about time: Exhaustive annotation of temporal data . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Gunnar A Sigurdsson, Olga Russakovsky, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Much ado about time: Exhaustive annotation of temporal data. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_51_1","volume-title":"highfrequency annotated camera trap images of 40 mammalian species in an African savanna. Scientifc data 2","author":"Swanson Alexandra","year":"2015","unstructured":"Alexandra Swanson , Margaret Kosmala , Chris Lintott , Robert Simpson , Arfon Smith , and Craig Packer . 2015. Snapshot Serengeti , highfrequency annotated camera trap images of 40 mammalian species in an African savanna. Scientifc data 2 ( 2015 ), 150026. Alexandra Swanson, Margaret Kosmala, Chris Lintott, Robert Simpson, Arfon Smith, and Craig Packer. 2015. Snapshot Serengeti, highfrequency annotated camera trap images of 40 mammalian species in an African savanna. Scientifc data 2 (2015), 150026."},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702420"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2013.25"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-012-0564-1"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1001220"},{"key":"e_1_3_2_2_56_1","volume-title":"Places: A 10 million image database for scene recognition","author":"Zhou Bolei","year":"2018","unstructured":"Bolei Zhou , Agata Lapedriza , Aditya Khosla , Aude Oliva , and Antonio Torralba . 2018 . Places: A 10 million image database for scene recognition . IEEE transactions on Pattern Analysis and Machine Intelligence 40, 6 (2018), 1452--1464. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE transactions on Pattern Analysis and Machine Intelligence 40, 6 (2018), 1452--1464."}],"event":{"name":"CHI '19: CHI Conference on Human Factors in Computing Systems","location":"Glasgow Scotland Uk","acronym":"CHI '19","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300522","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3290605.3300522","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3290605.3300522","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:32Z","timestamp":1750204412000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300522"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,2]]},"references-count":56,"alternative-id":["10.1145\/3290605.3300522","10.1145\/3290605"],"URL":"https:\/\/doi.org\/10.1145\/3290605.3300522","relation":{},"subject":[],"published":{"date-parts":[[2019,5,2]]},"assertion":[{"value":"2019-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}