{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:01:29Z","timestamp":1755838889888,"version":"3.41.0"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["IIS-1447416","IIS-1651969","IIS-1527453"],"award-info":[{"award-number":["IIS-1447416","IIS-1651969","IIS-1527453"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,11,7]]},"abstract":"<jats:p>Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses.<\/jats:p>","DOI":"10.1145\/3359238","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T13:40:21Z","timestamp":1573220421000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Dropping the Baton?"],"prefix":"10.1145","volume":"3","author":[{"given":"Tianyi","family":"Li","sequence":"first","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chandler J.","family":"Manns","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chris","family":"North","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kurt","family":"Luther","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"volume-title":"Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '14)","author":"Andr\u00e9 Paul","key":"e_1_2_1_1_1","unstructured":"Paul Andr\u00e9 , Aniket Kittur , and Steven P. Dow . 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data . In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '14) . ACM, New York, NY, USA, 989--998. https:\/\/doi.org\/10.1145\/2531602.2531653 10.1145\/2531602.2531653 Paul Andr\u00e9, Aniket Kittur, and Steven P. Dow. 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '14). ACM, New York, NY, USA, 989--998. https:\/\/doi.org\/10.1145\/2531602.2531653"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998294"},{"key":"e_1_2_1_3_1","volume-title":"Pay Attention","author":"August Tal","year":"2019","unstructured":"Tal August and Katharina Reinecke . 2019. Pay Attention , Please : Formal Language Improves Attention in Volunteer and Paid Online Experiments . ( 2019 ). Tal August and Katharina Reinecke. 2019. Pay Attention, Please: Formal Language Improves Attention in Volunteer and Paid Online Experiments. (2019)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2008.4677362"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187971"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2013.07.004"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16)","author":"Chan Joel","year":"1804","unstructured":"Joel Chan , Steven Dang , and Steven P. Dow . 2016. Improving Crowd Innovation with Expert Facilitation . In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16) . ACM, New York, NY, USA, 1223--1235. https:\/\/doi.org\/10.1145\/28 1804 8.2820023 10.1145\/2818048.2820023 Joel Chan, Steven Dang, and Steven P. Dow. 2016. Improving Crowd Innovation with Expert Facilitation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1223--1235. https:\/\/doi.org\/10.1145\/2818048.2820023"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858411"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1502650.1502668"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13)","author":"Chilton Lydia B.","year":"1999","unstructured":"Lydia B. Chilton , Greg Little , Darren Edge , Daniel S. Weld , and James A. Landay . 2013. Cascade: Crowdsourcing Taxonomy Creation . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13) . ACM, New York, NY, USA , 1999 --2008. https:\/\/doi.org\/10.1145\/2470654.2466265 10.1145\/2470654.2466265 Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing Taxonomy Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1999--2008. https:\/\/doi.org\/10.1145\/2470654.2466265"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09)","author":"Chin George","year":"1870","unstructured":"George Chin , Jr., Olga A. Kuchar , and Katherine E. Wolf . 2009. Exploring the Analytical Processes of Intelligence Analysts . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09) . ACM, New York, NY, USA, 11--20. https:\/\/doi.org\/10.1145\/15 1870 1.1518704 10.1145\/1518701.1518704 George Chin, Jr., Olga A. Kuchar, and Katherine E. Wolf. 2009. Exploring the Analytical Processes of Intelligence Analysts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 11--20. https:\/\/doi.org\/10.1145\/1518701.1518704"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2017.8585484"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2010.5652932"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866709"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207711"},{"key":"e_1_2_1_18_1","volume-title":"et almbox","author":"Gottlieb Steven","year":"1994","unstructured":"Steven Gottlieb , Sheldon I Arenberg , Raj Singh , et almbox . 1994 . Crime analysis: From first report to final arrest .Alpha Publishing Montclair , CA. Steven Gottlieb, Sheldon I Arenberg, Raj Singh, et almbox. 1994. Crime analysis: From first report to final arrest .Alpha Publishing Montclair, CA."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820071"},{"volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13)","author":"Goyal Nitesh","key":"e_1_2_1_20_1","unstructured":"Nitesh Goyal , Gilly Leshed , and Susan R. Fussell . 2013. Effects of Visualization and Note-taking on Sensemaking and Analysis . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13) . ACM, New York, NY, USA, 2721--2724. https:\/\/doi.org\/10.1145\/2470654.2481376 10.1145\/2470654.2481376 Nitesh Goyal, Gilly Leshed, and Susan R. Fussell. 2013. Effects of Visualization and Note-taking on Sensemaking and Analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2721--2724. https:\/\/doi.org\/10.1145\/2470654.2481376"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858364"},{"key":"e_1_2_1_22_1","unstructured":"Richards J Heuer. 1999. Psychology of intelligence analysis .Jeffrey Frank Jones.  Richards J Heuer. 1999. Psychology of intelligence analysis .Jeffrey Frank Jones."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470743"},{"key":"e_1_2_1_24_1","volume-title":"Groupthink: Psychological studies of policy decisions and fiascoes.","author":"Janis Irving Lester","year":"1982","unstructured":"Irving Lester Janis and Irving Lester Janis . 1982 . Groupthink: Psychological studies of policy decisions and fiascoes. Vol. 349 . Houghton Mifflin Boston . Irving Lester Janis and Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Vol. 349. Houghton Mifflin Boston."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531681"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998196"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2389176.2389219"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1807185116"},{"volume-title":"Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making.","author":"Klein Gary","key":"e_1_2_1_30_1","unstructured":"Gary Klein , Jennifer K Phillips , Erica L Rall , and Deborah A Peluso . 2007. A data-frame theory of sensemaking . In Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making. New York, NY, USA : Lawrence Erlbaum , 113--155. Gary Klein, Jennifer K Phillips, Erica L Rall, and Deborah A Peluso. 2007. A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making. New York, NY, USA: Lawrence Erlbaum, 113--155."},{"key":"e_1_2_1_31_1","volume-title":"Sixth AAAI Conference on Human Computation and Crowdsourcing .","author":"Kobayashi Masaki","year":"2018","unstructured":"Masaki Kobayashi , Hiromi Morita , Masaki Matsubara , Nobuyuki Shimizu , and Atsuyuki Morishima . 2018 . An empirical study on short-and long-term effects of self-correction in crowdsourced microtasks . In Sixth AAAI Conference on Human Computation and Crowdsourcing . Masaki Kobayashi, Hiromi Morita, Masaki Matsubara, Nobuyuki Shimizu, and Atsuyuki Morishima. 2018. An empirical study on short-and long-term effects of self-correction in crowdsourced microtasks. In Sixth AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15)","author":"Lasecki Walter S.","year":"1925","unstructured":"Walter S. Lasecki , Juho Kim , Nick Rafter , Onkur Sen , Jeffrey P. Bigham , and Michael S. Bernstein . 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15) . ACM, New York, NY, USA , 1925 --1934. https:\/\/doi.org\/10.1145\/2702123.2702565 10.1145\/2702123.2702565 Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1925--1934. https:\/\/doi.org\/10.1145\/2702123.2702565"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274374"},{"key":"e_1_2_1_35_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing .","author":"Luther Kurt","year":"2015","unstructured":"Kurt Luther , Nathan Hahn , Steven P Dow , and Aniket Kittur . 2015 . Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines . In Third AAAI Conference on Human Computation and Crowdsourcing . Kurt Luther, Nathan Hahn, Steven P Dow, and Aniket Kittur. 2015. Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines. In Third AAAI Conference on Human Computation and Crowdsourcing ."},{"volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '15)","author":"Luther Kurt","key":"e_1_2_1_36_1","unstructured":"Kurt Luther , Jari-Lee Tolentino , Wei Wu , Amy Pavel , Brian P. Bailey , Maneesh Agrawala , Bj\u00f6rn Hartmann , and Steven P. Dow . 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique . In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '15) . ACM, New York, NY, USA, 473--485. https:\/\/doi.org\/10.1145\/2675133.2675283 10.1145\/2675133.2675283 Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Bj\u00f6rn Hartmann, and Steven P. Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work &#38; Social Computing (CSCW '15). ACM, New York, NY, USA, 473--485. https:\/\/doi.org\/10.1145\/2675133.2675283"},{"key":"#cr-split#-e_1_2_1_37_1.1","doi-asserted-by":"crossref","unstructured":"Filippo Menczer. 2016. The Spread of Misinformation in Social Media. In Proceedings of the 25th International Conference Companion on World Wide Web (WWW '16 Companion). International World Wide Web Conferences Steering Committee Republic and Canton of Geneva Switzerland 717--717. https:\/\/doi.org\/10.1145\/2872518.2890092 10.1145\/2872518.2890092","DOI":"10.1145\/2872518.2890092"},{"key":"#cr-split#-e_1_2_1_37_1.2","doi-asserted-by":"crossref","unstructured":"Filippo Menczer. 2016. The Spread of Misinformation in Social Media. In Proceedings of the 25th International Conference Companion on World Wide Web (WWW '16 Companion). International World Wide Web Conferences Steering Committee Republic and Canton of Geneva Switzerland 717--717. https:\/\/doi.org\/10.1145\/2872518.2890092","DOI":"10.1145\/2872518.2890092"},{"key":"e_1_2_1_38_1","volume-title":"ACM Conference on Collective Intelligence (CI 2018)","author":"Mohanty Vikram","year":"2018","unstructured":"Vikram Mohanty , David Thames , and Kurt Luther . 2018 . Photo Sleuth: Combining Collective Intelligence and Computer Vision to Identify Historical Portraits . In ACM Conference on Collective Intelligence (CI 2018) . Vikram Mohanty, David Thames, and Kurt Luther. 2018. Photo Sleuth: Combining Collective Intelligence and Computer Vision to Identify Historical Portraits. In ACM Conference on Collective Intelligence (CI 2018) ."},{"key":"e_1_2_1_39_1","volume-title":"Subcontracting Microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17)","author":"Morris Meredith Ringel","year":"2017","unstructured":"Meredith Ringel Morris , Jeffrey P. Bigham , Robin Brewer , Jonathan Bragg , Anand Kulkarni , Jessie Li , and Saiph Savage . 2017 . Subcontracting Microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA , 1867--1876. https:\/\/doi.org\/10.1145\/3025453.3025687 10.1145\/3025453.3025687 Meredith Ringel Morris, Jeffrey P. Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting Microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1867--1876. https:\/\/doi.org\/10.1145\/3025453.3025687"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025687"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/acp.3213"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of international conference on intelligence analysis","volume":"5","author":"Pirolli Peter","year":"2005","unstructured":"Peter Pirolli and Stuart Card . 2005 . The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis . In Proceedings of international conference on intelligence analysis , Vol. 5 . McLean, VA, USA, 2--4. Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2--4."},{"key":"e_1_2_1_43_1","first-page":"23 pages. https","volume-title":"No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. Proc. ACM Hum.-Comput. Interact.","volume":"1","author":"Retelny Daniela","year":"2017","unstructured":"Daniela Retelny , Michael S. Bernstein , and Melissa A. Valentine . 2017 . No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. Proc. ACM Hum.-Comput. Interact. , Vol. 1 , CSCW, Article 89 ( Dec. 2017 ), bibinfonumpages 23 pages. https:\/\/doi.org\/10 .1145\/3134724 10.1145\/3134724 Daniela Retelny, Michael S. Bernstein, and Melissa A. Valentine. 2017. No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. Proc. ACM Hum.-Comput. Interact., Vol. 1, CSCW, Article 89 (Dec. 2017), bibinfonumpages23 pages. https:\/\/doi.org\/10.1145\/3134724"},{"key":"e_1_2_1_44_1","article-title":"Online Sequencing of Non-Decomposable Macrotasks in Expert","volume":"1","author":"Schmitz Heinz","year":"2018","unstructured":"Heinz Schmitz and Ioanna Lykourentzou . 2018 . Online Sequencing of Non-Decomposable Macrotasks in Expert Crowdsourcing. Trans. Soc. Comput. , Vol. 1 , 1, Article 1 (Jan. 2018), bibinfonumpages33 pages. https:\/\/doi.org\/10.1145\/3140459 10.1145\/3140459 Heinz Schmitz and Ioanna Lykourentzou. 2018. Online Sequencing of Non-Decomposable Macrotasks in Expert Crowdsourcing. Trans. Soc. Comput., Vol. 1, 1, Article 1 (Jan. 2018), bibinfonumpages33 pages. https:\/\/doi.org\/10.1145\/3140459","journal-title":"Crowdsourcing. Trans. Soc. Comput."},{"key":"e_1_2_1_45_1","volume-title":"Sensemaking Handoffs: Why? How? and When?","author":"Sharma Nikhil","year":"2010","unstructured":"Nikhil Sharma . 2010 . Sensemaking Handoffs: Why? How? and When? (2010). Nikhil Sharma. 2010. Sensemaking Handoffs: Why? How? and When? (2010)."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557337"},{"key":"e_1_2_1_47_1","volume-title":"Biset: Semantic edge bundling with biclusters for sensemaking","author":"Sun Maoyuan","year":"2015","unstructured":"Maoyuan Sun , Peng Mi , Chris North , and Naren Ramakrishnan . 2015 . Biset: Semantic edge bundling with biclusters for sensemaking . IEEE transactions on visualization and computer graphics, Vol. 22 , 1 (2015), 310--319. Maoyuan Sun, Peng Mi, Chris North, and Naren Ramakrishnan. 2015. Biset: Semantic edge bundling with biclusters for sensemaking. IEEE transactions on visualization and computer graphics, Vol. 22, 1 (2015), 310--319."},{"key":"e_1_2_1_48_1","volume-title":"Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference on Web and Social Media .","author":"Tausczik Yla","year":"2018","unstructured":"Yla Tausczik and Mark Boons . 2018 . Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference on Web and Social Media . Yla Tausczik and Mark Boons. 2018. Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference on Web and Social Media ."},{"volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17)","author":"Valentine Melissa A.","key":"e_1_2_1_49_1","unstructured":"Melissa A. Valentine , Daniela Retelny , Alexandra To , Negar Rahmati , Tulsee Doshi , and Michael S. Bernstein . 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA, 3523--3537. https:\/\/doi.org\/10.1145\/3025453.3025811 10.1145\/3025453.3025811 Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3523--3537. https:\/\/doi.org\/10.1145\/3025453.3025811"},{"key":"e_1_2_1_50_1","volume-title":"Second AAAI Conference on Human Computation and Crowdsourcing .","author":"Verroios Vasilis","year":"2014","unstructured":"Vasilis Verroios and Michael S Bernstein . 2014 . Context trees: Crowdsourcing global understanding from local views . In Second AAAI Conference on Human Computation and Crowdsourcing . Vasilis Verroios and Michael S Bernstein. 2014. Context trees: Crowdsourcing global understanding from local views. In Second AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207709"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124890"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2018.2877733"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16)","author":"Yu Lixiu","year":"1804","unstructured":"Lixiu Yu , Aniket Kittur , and Robert E. Kraut . 2016. Encouraging \"Outside- The- Box\" Thinking in Crowd Innovation Through Identifying Domains of Expertise . In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16) . ACM, New York, NY, USA, 1214--1222. https:\/\/doi.org\/10.1145\/28 1804 8.2820025 10.1145\/2818048.2820025 Lixiu Yu, Aniket Kittur, and Robert E. Kraut. 2016. Encouraging \"Outside- The- Box\" Thinking in Crowd Innovation Through Identifying Domains of Expertise. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1214--1222. https:\/\/doi.org\/10.1145\/2818048.2820025"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"},{"key":"e_1_2_1_56_1","volume-title":"Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs","author":"Zhao Jian","year":"2017","unstructured":"Jian Zhao , Michael Glueck , Petra Isenberg , Fanny Chevalier , and Azam Khan . 2017. Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 ( 2017 ), 340--350. Jian Zhao, Michael Glueck, Petra Isenberg, Fanny Chevalier, and Azam Khan. 2017. Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 340--350."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359238","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359238","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359238","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:54Z","timestamp":1750201974000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359238"}},"subtitle":["Understanding Errors and Bottlenecks in a Crowdsourced Sensemaking Pipeline"],"short-title":[],"issued":{"date-parts":[[2019,11,7]]},"references-count":57,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2019,11,7]]}},"alternative-id":["10.1145\/3359238"],"URL":"https:\/\/doi.org\/10.1145\/3359238","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2019,11,7]]},"assertion":[{"value":"2019-11-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}