{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:51:47Z","timestamp":1772592707273,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":39,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,3,5]],"date-time":"2018-03-05T00:00:00Z","timestamp":1520208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,3,5]]},"DOI":"10.1145\/3172944.3172948","type":"proceedings-article","created":{"date-parts":[[2018,3,9]],"date-time":"2018-03-09T13:28:34Z","timestamp":1520602114000},"page":"559-570","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Two Tools are Better Than One"],"prefix":"10.1145","author":[{"given":"Jean Y.","family":"Song","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}]},{"given":"Raymond","family":"Fok","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}]},{"given":"Alan","family":"Lundgard","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor , MI, USA"}]},{"given":"Fan","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}]},{"given":"Juho","family":"Kim","sequence":"additional","affiliation":[{"name":"Korea Advanced Institute of Science and Technology, Daejeon, Rebublic of Korea"}]},{"given":"Walter S.","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,3,5]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_34"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462002"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791285"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660125"},{"key":"e_1_3_2_1_5_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics ( 1979 ), 20--28. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979), 20--28."},{"key":"e_1_3_2_1_6_1","volume-title":"Ensemble methods in machine learning. Multiple classifier systems 1857","author":"Dietterich Thomas G","year":"2000","unstructured":"Thomas G Dietterich and others. 2000. Ensemble methods in machine learning. Multiple classifier systems 1857 ( 2000 ), 1--15. Thomas G Dietterich and others. 2000. Ensemble methods in machine learning. Multiple classifier systems 1857 (2000), 1--15."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/646943.712093"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025930"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Mitchell Gordon Jeffrey P Bigham and Walter S Lasecki. 2015. LegionTools: a toolkit  Mitchell Gordon Jeffrey P Bigham and Walter S Lasecki. 2015. LegionTools: a toolkit","DOI":"10.1145\/2815585.2815729"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815585.2815729"},{"key":"e_1_3_2_1_12_1","volume-title":"Towards Hybrid Intelligence for Robotics. Collective Intelligence Conference (CI)","author":"Gouravajhala Sai","year":"2017","unstructured":"Sai Gouravajhala , Jean Y Song , Jinyeong Yim , Raymond Fok , Yanda Huang , Fan Yang , Kyle Wang , Yilei An , and Walter S Lasecki . 2017 . Towards Hybrid Intelligence for Robotics. Collective Intelligence Conference (CI) (2017). Sai Gouravajhala, Jean Y Song, Jinyeong Yim, Raymond Fok, Yanda Huang, Fan Yang, Kyle Wang, Yilei An, and Walter S Lasecki. 2017. Towards Hybrid Intelligence for Robotics. Collective Intelligence Conference (CI) (2017)."},{"key":"e_1_3_2_1_13_1","volume-title":"Investigating the Influence of Data Familiarity to Improve the Design of a Crowdsourcing Image Annotation System. HCOMP","author":"Gurari Danna","year":"2016","unstructured":"Danna Gurari , Mehrnoosh Sameki , and Margrit Betke . 2016. Investigating the Influence of Data Familiarity to Improve the Design of a Crowdsourcing Image Annotation System. HCOMP ( 2016 ). Danna Gurari, Mehrnoosh Sameki, and Margrit Betke. 2016. Investigating the Influence of Data Familiarity to Improve the Design of a Crowdsourcing Image Annotation System. HCOMP (2016)."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.58871"},{"key":"e_1_3_2_1_15_1","volume-title":"Girshick","author":"He Kaiming","year":"2017","unstructured":"Kaiming He , Georgia Gkioxari , Piotr Doll\u00e1r , and Ross B . Girshick . 2017 . Mask R-CNN. CoRR abs\/1703.06870 (2017). Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross B. Girshick. 2017. Mask R-CNN. CoRR abs\/1703.06870 (2017)."},{"key":"e_1_3_2_1_16_1","volume-title":"First AAAI conference on human computation and crowdsourcing.","author":"Bragg Mausam Jonathan","unstructured":"Mausam Jonathan Bragg and Daniel S. Weld . 2013. Crowdsourcing multi-label classification for taxonomy creation . In First AAAI conference on human computation and crowdsourcing. Mausam Jonathan Bragg and Daniel S. Weld. 2013. Crowdsourcing multi-label classification for taxonomy creation. In First AAAI conference on human computation and crowdsourcing."},{"key":"e_1_3_2_1_17_1","volume-title":"Proceedings of the AAAI Conference on Human Computation (HCOMP","volume":"17","author":"Kaur Harmanpreet","year":"2017","unstructured":"Harmanpreet Kaur , Mitchell Gordon , Yiwei Yang , Jeffrey P Bigham , Jaime Teevan , Ece Kamar , and Walter S Lasecki . 2017 . Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering . In Proceedings of the AAAI Conference on Human Computation (HCOMP 2017)., HCOMP, Vol. 17 . Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P Bigham, Jaime Teevan, Ece Kamar, and Walter S Lasecki. 2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation (HCOMP 2017)., HCOMP, Vol. 17."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2556986"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647367"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466269"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047200"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441912"},{"key":"e_1_3_2_1_24_1","volume-title":"Mechanical turk is not anonymous. Social Science Research Network (SSRN)","author":"Lease Matthew","year":"2013","unstructured":"Matthew Lease , Jessica Hullman , Jeffrey P Bigham , Michael S Bernstein , Juho Kim , Walter Lasecki , Saeideh Bakhshi , Tanushree Mitra , and Robert C Miller . 2013. Mechanical turk is not anonymous. Social Science Research Network (SSRN) ( 2013 ). Matthew Lease, Jessica Hullman, Jeffrey P Bigham, Michael S Bernstein, Juho Kim, Walter Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C Miller. 2013. Mechanical turk is not anonymous. Social Science Research Network (SSRN) (2013)."},{"key":"e_1_3_2_1_25_1","volume-title":"In: Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, UAI.","author":"Lin Christopher H.","unstructured":"Christopher H. Lin , Mausam Daniel , and S. Weld . 2012. Crowdsourcing control: Moving beyond multiple choice . In In: Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, UAI. Christopher H. Lin, Mausam Daniel, and S. Weld. 2012. Crowdsourcing control: Moving beyond multiple choice. In In: Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, UAI."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.344"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_1_28_1","volume-title":"Fully Convolutional Networks for Semantic Segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Long Jonathan","year":"2015","unstructured":"Jonathan Long , Evan Shelhamer , and Trevor Darrell . 2015 . Fully Convolutional Networks for Semantic Segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully Convolutional Networks for Semantic Segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_3_2_1_29_1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)","author":"Lundgard Alan","unstructured":"Alan Lundgard , Yiwei Yang , Maya L. Foster , and Walter S. Lasecki . 2018. Bolt: Instantaneous Crowdsourcing via Just-in-Time Training . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18) . ACM, New York, NY, USA. Alan Lundgard, Yiwei Yang, Maya L. Foster, and Walter S. Lasecki. 2018. Bolt: Instantaneous Crowdsourcing via Just-in-Time Training. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0603%264_2"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v1i1.13075"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208695"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298824"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-007-0090-8"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126603"},{"key":"e_1_3_2_1_37_1","volume-title":"Applied computer science","author":"Torbert Shane","unstructured":"Shane Torbert . 2016. Applied computer science . Springer . Shane Torbert. 2016. Applied computer science. Springer."},{"key":"e_1_3_2_1_38_1","volume-title":"Belongie","author":"Welinder Peter","year":"2010","unstructured":"Peter Welinder , Steve Branson , Pietro Perona , and Serge J . Belongie . 2010 . The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems 23, J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (Eds.). Curran Associates, Inc ., 2424--2432. Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. 2010. The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems 23, J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (Eds.). Curran Associates, Inc., 2424--2432."},{"key":"e_1_3_2_1_39_1","volume-title":"Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo.","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting fan Wu , Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009 . Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc ., 2035--2043. Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc., 2035--2043."}],"event":{"name":"IUI'18: 23rd International Conference on Intelligent User Interfaces","location":"Tokyo Japan","acronym":"IUI'18","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["23rd International Conference on Intelligent User Interfaces"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3172944.3172948","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3172944.3172948","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:15:18Z","timestamp":1750281318000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3172944.3172948"}},"subtitle":["Tool Diversity as a Means of Improving Aggregate Crowd Performance"],"short-title":[],"issued":{"date-parts":[[2018,3,5]]},"references-count":39,"alternative-id":["10.1145\/3172944.3172948","10.1145\/3172944"],"URL":"https:\/\/doi.org\/10.1145\/3172944.3172948","relation":{},"subject":[],"published":{"date-parts":[[2018,3,5]]},"assertion":[{"value":"2018-03-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}