{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T05:29:14Z","timestamp":1761629354550,"version":"3.41.0"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,5,5]],"date-time":"2016-05-05T00:00:00Z","timestamp":1462406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/G066051\/1"],"award-info":[{"award-number":["EP\/G066051\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,7,14]]},"abstract":"<jats:p>\n            We present an incremental Bayesian model that resolves key issues of crowd size and data quality for consensus labeling. We evaluate our method using data collected from a real-world citizen science program, B\n            <jats:sc>ee<\/jats:sc>\n            W\n            <jats:sc>atch<\/jats:sc>\n            , which invites members of the public in the United Kingdom to classify (label) photographs of bumblebees as one of 22 possible species. The biological recording domain poses two key and hitherto unaddressed challenges for consensus models of crowdsourcing: (1) the large number of potential species makes classification difficult, and (2) this is compounded by limited crowd availability, stemming from both the inherent difficulty of the task and the lack of relevant skills among the general public. We demonstrate that consensus labels can be reliably found in such circumstances with very small crowd sizes of around three to five users (i.e., through group sourcing). Our incremental Bayesian model, which minimizes crowd size by re-evaluating the quality of the consensus label following each species identification solicited from the crowd, is competitive with a Bayesian approach that uses a larger but fixed crowd size and outperforms majority voting. These results have important ecological applicability: biological recording programs such as B\n            <jats:sc>ee<\/jats:sc>\n            W\n            <jats:sc>atch<\/jats:sc>\n            can sustain themselves when resources such as taxonomic experts to confirm identifications by photo submitters are scarce (as is typically the case), and feedback can be provided to submitters in a timely fashion. More generally, our model provides benefits to any crowdsourced consensus labeling task where there is a cost (financial or otherwise) associated with soliciting a label.\n          <\/jats:p>","DOI":"10.1145\/2776896","type":"journal-article","created":{"date-parts":[[2016,5,6]],"date-time":"2016-05-06T12:59:12Z","timestamp":1462539552000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Crowdsourcing Without a Crowd"],"prefix":"10.1145","volume":"7","author":[{"given":"Advaith","family":"Siddharthan","sequence":"first","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher","family":"Lambin","sequence":"additional","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anne-Marie","family":"Robinson","sequence":"additional","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nirwan","family":"Sharma","sequence":"additional","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"Comont","sequence":"additional","affiliation":[{"name":"Bumblebee Conservation Trust, Stirling, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elaine","family":"O'mahony","sequence":"additional","affiliation":[{"name":"Bumblebee Conservation Trust, Stirling, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chris","family":"Mellish","sequence":"additional","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ren\u00e9 Van Der","family":"Wal","sequence":"additional","affiliation":[{"name":"University of Aberdeen, Aberdeen, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,5,5]]},"reference":[{"volume-title":"Proceedings of the 24th International Conference on Computational Linguistics (COLING\u201912)","author":"Blake Steven","key":"e_1_2_2_1_1","unstructured":"Steven Blake , Advaith Siddharthan , Hien Nguyen , Nirwan Sharma , Anne-Marie Robinson , Elaine O\u2019Mahony , Ben Darvill , Chris Mellish , and Ren\u00e9 van der Wal. 2012. Natural language generation for nature conservation: Automating feedback to help volunteers identify bumblebee species . In Proceedings of the 24th International Conference on Computational Linguistics (COLING\u201912) . 311--324. Steven Blake, Advaith Siddharthan, Hien Nguyen, Nirwan Sharma, Anne-Marie Robinson, Elaine O\u2019Mahony, Ben Darvill, Chris Mellish, and Ren\u00e9 van der Wal. 2012. Natural language generation for nature conservation: Automating feedback to help volunteers identify bumblebee species. In Proceedings of the 24th International Conference on Computational Linguistics (COLING\u201912). 311--324."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1890\/110273"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090116.2090121"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2699.2012.02734.x"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10531-005-8375-0"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458165"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10336-007-0239-9"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tree.2011.11.006"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems","volume":"1","author":"Kamar Ece","year":"2012","unstructured":"Ece Kamar , Severin Hacker , and Eric Horvitz . 2012 . Combining human and machine intelligence in large-scale crowdsourcing . In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems , Volume 1 . 467--474. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, Volume 1. 467--474."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2013.1235"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568033"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2966.2008.13689.x"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600159"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660123"},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing.","author":"Mao Andrew","year":"2013","unstructured":"Andrew Mao , Ece Kamar , and Eric Horvitz . 2013 . Why stop now? Predicting worker engagement in online crowdsourcing . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing. Andrew Mao, Ece Kamar, and Eric Horvitz. 2013. Why stop now? Predicting worker engagement in online crowdsourcing. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1472-4642.2012.00883.x"},{"volume-title":"Science for Environment Policy Indepth Report: Environmental Citizen Science. Report produced for the European Commission DG Environment. European Commission","author":"Unit Science Communication","key":"e_1_2_2_20_1","unstructured":"Science Communication Unit . 2013. Science for Environment Policy Indepth Report: Environmental Citizen Science. Report produced for the European Commission DG Environment. European Commission , University of the West of England , Bristol. Available at http:\/\/ec.europa.eu\/science-environment-policy. Science Communication Unit. 2013. Science for Environment Policy Indepth Report: Environmental Citizen Science. Report produced for the European Commission DG Environment. European Commission, University of the West of England, Bristol. Available at http:\/\/ec.europa.eu\/science-environment-policy."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_2_22_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing.","author":"Sheshadri Aashish","year":"2013","unstructured":"Aashish Sheshadri and Matthew Lease . 2013 . SQUARE: A benchmark for research on computing crowd consensus . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A benchmark for research on computing crowd consensus. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.3897\/zookeys.480.8803"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/0378-3758(94)00121-B"},{"volume-title":"Proceedings of the the Conference on Empirical Methods in Natural Language Processing. 254--263","author":"Snow Rion","key":"e_1_2_2_25_1","unstructured":"Rion Snow , Brendan O\u2019Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks . In Proceedings of the the Conference on Empirical Methods in Natural Language Processing. 254--263 . Rion Snow, Brendan O\u2019Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the the Conference on Empirical Methods in Natural Language Processing. 254--263."},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the Workshop on Crowdsourcing for Information Retrieval (CIR\u201911)","author":"Tang Wei","year":"2011","unstructured":"Wei Tang and Matthew Lease . 2011 . Semi-supervised consensus labeling for crowdsourcing . In Proceedings of the Workshop on Crowdsourcing for Information Retrieval (CIR\u201911) . Wei Tang and Matthew Lease. 2011. Semi-supervised consensus labeling for crowdsourcing. In Proceedings of the Workshop on Crowdsourcing for Information Retrieval (CIR\u201911)."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13280-015-0709-x"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_2_2_29_1","volume-title":"Ruvolo","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-Fan Wu , Jacob Bergsma , Javier R. Movellan , and Paul L . Ruvolo . 2009 . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 (NIPS\u2019 09). 2035--2043. Jacob Whitehill, Ting-Fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 (NIPS\u201909). 2035--2043."},{"volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML-11)","author":"Yan Yan","key":"e_1_2_2_30_1","unstructured":"Yan Yan , Glenn M. Fung , R\u00f3mer Rosales , and Jennifer G. Dy . 2011. Active learning from crowds . In Proceedings of the 28th International Conference on Machine Learning (ICML-11) . 1161--1168. Yan Yan, Glenn M. Fung, R\u00f3mer Rosales, and Jennifer G. Dy. 2011. Active learning from crowds. In Proceedings of the 28th International Conference on Machine Learning (ICML-11). 1161--1168."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2776896","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2776896","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:04:01Z","timestamp":1750273441000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2776896"}},"subtitle":["Reliable Online Species Identification Using Bayesian Models to Minimize Crowd Size"],"short-title":[],"issued":{"date-parts":[[2016,5,5]]},"references-count":30,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,14]]}},"alternative-id":["10.1145\/2776896"],"URL":"https:\/\/doi.org\/10.1145\/2776896","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2016,5,5]]},"assertion":[{"value":"2015-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-05-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}