{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:26:57Z","timestamp":1772591217212,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,8,9]],"date-time":"2019-08-09T00:00:00Z","timestamp":1565308800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Institute for Information 8 communications Technology Promotio"},{"name":"Development of Autonomous IoT Collaboration Framework for Space Intelligence","award":["2017-0-00537"],"award-info":[{"award-number":["2017-0-00537"]}]},{"name":"Korea government"},{"name":"Denso Corporation, Toyota Research Institute, and MCity at the University of Michigan"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2020,3,31]]},"abstract":"<jats:p>\n            Crowdsourcing is a common means of collecting image segmentation training data for use in a variety of computer vision applications. However, designing accurate crowd-powered image segmentation systems is challenging, because defining object boundaries in an image requires significant fine motor skills and hand-eye coordination, which makes these tasks error-prone. Typically, special segmentation tools are created and then answers from multiple workers are aggregated to generate more accurate results. However, individual tool designs can bias how and where people make mistakes, resulting in shared errors that remain even after aggregation. In this article, we introduce a novel crowdsourcing approach that leverages\n            <jats:italic>tool diversity<\/jats:italic>\n            as a means of improving aggregate crowd performance. Our idea is that given a diverse set of tools, answer aggregation done\n            <jats:italic>across<\/jats:italic>\n            tools can help improve the collective performance by offsetting systematic biases induced by the individual tools themselves. To demonstrate the effectiveness of the proposed approach, we design four different tools and present FourEyes, a crowd-powered image segmentation system that uses aggregation across different tools. We then conduct a series of studies that evaluate different aggregation conditions and show that using multiple tools can significantly improve aggregate accuracy. Furthermore, we investigate the idea of applying post-processing for multi-tool aggregation in terms of correction mechanism. We introduce a novel region-based method for synthesizing more accurate bounds for image segmentation tasks through averaging surrounding annotations. In addition, we explore the effect of adjusting the threshold parameter of an EM-based aggregation method. Our results suggest that not only the individual tool\u2019s design, but also the correction mechanism, can affect the performance of multi-tool aggregation. This article extends a work presented at ACM IUI 2018\u00a0[46] by providing a novel region-based error-correction method and additional in-depth evaluation of the proposed approach.\n          <\/jats:p>","DOI":"10.1145\/3237188","type":"journal-article","created":{"date-parts":[[2019,8,9]],"date-time":"2019-08-09T12:22:28Z","timestamp":1565353348000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["FourEyes"],"prefix":"10.1145","volume":"10","author":[{"given":"Jean Y.","family":"Song","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raymond","family":"Fok","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juho","family":"Kim","sequence":"additional","affiliation":[{"name":"KAIST, Republic of Korea, Daejeon, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Walter S.","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,8,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_34"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462002"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing.","author":"Bragg Jonathan","unstructured":"Jonathan Bragg , Mausam, and Daniel S. Weld . 2013. Crowdsourcing multi-label classification for taxonomy creation . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing. Jonathan Bragg, Mausam, and Daniel S. Weld. 2013. Crowdsourcing multi-label classification for taxonomy creation. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660125"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/648054.743935"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_9_1","volume-title":"Schapire","author":"Freund Yoav","year":"1995","unstructured":"Yoav Freund and Robert E . Schapire . 1995 . A desicion-theoretic generalization of on-line learning and an application to boosting. In Proceedings of the European Conference on Computational Learning Theory. Springer , 23--37. Yoav Freund and Robert E. Schapire. 1995. A desicion-theoretic generalization of on-line learning and an application to boosting. In Proceedings of the European Conference on Computational Learning Theory. Springer, 23--37."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025930"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815585.2815729"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Collective Intelligence Conference (CI\u201917)","author":"Gouravajhala Sai","unstructured":"Sai Gouravajhala , Jean Y. Song , Jinyeong Yim , Raymond Fok , Yanda Huang , Fan Yang , Kyle Wang , Yilei An , and Walter S. Lasecki . 2017. Towards hybrid intelligence for robotics . In Proceedings of the Collective Intelligence Conference (CI\u201917) . Sai Gouravajhala, Jean Y. Song, Jinyeong Yim, Raymond Fok, Yanda Huang, Fan Yang, Kyle Wang, Yilei An, and Walter S. Lasecki. 2017. Towards hybrid intelligence for robotics. In Proceedings of the Collective Intelligence Conference (CI\u201917)."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the AAAI Conference on Human Computation 8 Crowdsourcing (HCOMP\u201916)","author":"Gurari Danna","year":"2016","unstructured":"Danna Gurari , Mehrnoosh Sameki , and Margrit Betke . 2016 . Investigating the influence of data familiarity to improve the design of a crowdsourcing image annotation system . In Proceedings of the AAAI Conference on Human Computation 8 Crowdsourcing (HCOMP\u201916) . Danna Gurari, Mehrnoosh Sameki, and Margrit Betke. 2016. Investigating the influence of data familiarity to improve the design of a crowdsourcing image annotation system. In Proceedings of the AAAI Conference on Human Computation 8 Crowdsourcing (HCOMP\u201916)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.58871"},{"key":"e_1_2_1_15_1","volume-title":"Girshick","author":"He Kaiming","year":"2017","unstructured":"Kaiming He , Georgia Gkioxari , Piotr Doll\u00e1r , and Ross B . Girshick . 2017 . Mask R-CNN. Retrieved from: CoRR abs\/1703.06870. Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross B. Girshick. 2017. Mask R-CNN. Retrieved from: CoRR abs\/1703.06870."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173685"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the AAAI Conference on Human Computation (HCOMP\u201917)","volume":"17","author":"Kaur Harmanpreet","unstructured":"Harmanpreet Kaur , Mitchell Gordon , Yiwei Yang , Jeffrey P. Bigham , Jaime Teevan , Ece Kamar , and Walter S. Lasecki . 2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering . In Proceedings of the AAAI Conference on Human Computation (HCOMP\u201917) , Vol. 17 . Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P. Bigham, Jaime Teevan, Ece Kamar, and Walter S. Lasecki. 2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation (HCOMP\u201917), Vol. 17."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2556986"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2212776.2223835"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647367"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466269"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047200"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441912"},{"key":"e_1_2_1_28_1","volume-title":"Miller","author":"Lease Matthew","year":"2013","unstructured":"Matthew Lease , Jessica Hullman , Jeffrey P. Bigham , Michael S. Bernstein , Juho Kim , Walter S. Lasecki , Saeideh Bakhshi , Tanushree Mitra , and Robert C . Miller . 2013 . Mechanical Turk is not anonymous. Soc. Sci. Res. Netw . (2013). Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller. 2013. Mechanical Turk is not anonymous. Soc. Sci. Res. Netw. (2013)."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence.","author":"Lin Christopher","year":"2012","unstructured":"Christopher Lin , Mausam Mausam , and Daniel Weld . 2012 . Dynamically switching between synergistic workflows for crowdsourcing . In Proceedings of the AAAI Conference on Artificial Intelligence. Christopher Lin, Mausam Mausam, and Daniel Weld. 2012. Dynamically switching between synergistic workflows for crowdsourcing. In Proceedings of the AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI\u201912)","author":"Lin Christopher H.","unstructured":"Christopher H. Lin , Mausam Daniel , and S. Weld . 2012. Crowdsourcing control: Moving beyond multiple choice . In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI\u201912) . Christopher H. Lin, Mausam Daniel, and S. Weld. 2012. Crowdsourcing control: Moving beyond multiple choice. In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI\u201912)."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.344"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the European Conference on Computer Vision. Springer, 740--755","author":"Lin Tsung-Yi","unstructured":"Tsung-Yi Lin , Michael Maire , Serge Belongie , James Hays , Pietro Perona , Deva Ramanan , Piotr Doll\u00e1r , and C. Lawrence Zitnick . 2014. Microsoft COCO: Common objects in context . In Proceedings of the European Conference on Computer Vision. Springer, 740--755 . Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision. Springer, 740--755."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173961"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174041"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing.","author":"Luther Kurt","year":"2015","unstructured":"Kurt Luther , Nathan Hahn , Steven P. Dow , and Aniket Kittur . 2015 . Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines . In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing. Kurt Luther, Nathan Hahn, Steven P. Dow, and Aniket Kittur. 2015. Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0603%264_2"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing.","author":"Mao Andrew","unstructured":"Andrew Mao , Ece Kamar , Yiling Chen , Eric Horvitz , Megan E. Schwamb , Chris J. Lintott , and Arfon M. Smith . 2013. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing . In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing. Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan E. Schwamb, Chris J. Lintott, and Arfon M. Smith. 2013. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/1076-8971.7.1.3"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208695"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the AAAI Conference on Human Computation. ACM.","author":"Rao Akshay","unstructured":"Akshay Rao , Harmanpreet Kaur , and Walter S. Lasecki . 2018. Plexiglass: Multiplexing passive and active tasks for more efficient crowdsourcing . In Proceedings of the AAAI Conference on Human Computation. ACM. Akshay Rao, Harmanpreet Kaur, and Walter S. Lasecki. 2018. Plexiglass: Multiplexing passive and active tasks for more efficient crowdsourcing. In Proceedings of the AAAI Conference on Human Computation. ACM."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-007-0090-8"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 254--263","author":"Snow Rion","unstructured":"Rion Snow , Brendan O\u2019Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and fast\u2014but is it good?: Evaluating non-expert annotations for natural language tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 254--263 . Rion Snow, Brendan O\u2019Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast\u2014but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 254--263."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172948"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126603"},{"key":"e_1_2_1_48_1","volume-title":"Applied Computer Science","author":"Torbert Shane","unstructured":"Shane Torbert . 2016. Applied Computer Science . Springer . 158 pages. Shane Torbert. 2016. Applied Computer Science. Springer. 158 pages."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 2424--2432","author":"Welinder Peter","unstructured":"Peter Welinder , Steve Branson , Pietro Perona , and Serge J. Belongie . 2010. The multidimensional wisdom of crowds . In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 2424--2432 . Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. 2010. The multidimensional wisdom of crowds. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 2424--2432."},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc.","author":"Whitehill Jacob","unstructured":"Jacob Whitehill , Ting Fan Wu , Jacob Bergsma , Javier R. Movellan , and Paul L. Ruvolo . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc. , 2035--2043. Jacob Whitehill, Ting Fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 2035--2043."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876042"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3237188","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3237188","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:39:34Z","timestamp":1750210774000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3237188"}},"subtitle":["Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing"],"short-title":[],"issued":{"date-parts":[[2019,8,9]]},"references-count":51,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,3,31]]}},"alternative-id":["10.1145\/3237188"],"URL":"https:\/\/doi.org\/10.1145\/3237188","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8,9]]},"assertion":[{"value":"2018-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-08-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}