{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T04:59:07Z","timestamp":1755838747820,"version":"3.41.0"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["IIS-1447416","IIS-1651969","IIS-1527453"],"award-info":[{"award-number":["IIS-1447416","IIS-1651969","IIS-1527453"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>The increasing volume of text data is challenging the cognitive capabilities of expert analysts. Machine learning and crowdsourcing present new opportunities for large-scale sensemaking, but we must overcome the challenge of modeling the overall process so that many distributed agents can contribute to suitable components asynchronously and meaningfully. In this paper, we explore how to crowdsource the sensemaking process via a pipeline of modularized steps connected by clearly defined inputs and outputs. Our pipeline restructures and partitions information into \"context slices\" for individual workers. We implemented CrowdIA, a software platform to enable unsupervised crowd sensemaking using our pipeline. With CrowdIA, crowds successfully solved two mysteries, and were one step away from solving the third. The crowd's intermediate results revealed their reasoning process and provided evidence that justifies their conclusions. We suggest broader possibilities to optimize each component, as well as to evaluate and refine previous intermediate analyses to improve the final result.<\/jats:p>","DOI":"10.1145\/3274374","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["CrowdIA"],"prefix":"10.1145","volume":"2","author":[{"given":"Tianyi","family":"Li","sequence":"first","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Kurt","family":"Luther","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Chris","family":"North","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531653"},{"key":"e_1_2_1_2_1","first-page":"135","article-title":"VisTrails: enabling interactive multiple-view visualizations. In VIS 05","volume":"2005","author":"Bavoil L.","year":"2005","unstructured":"L. Bavoil, S. P. Callahan, P. J. Crossno, J. Freire, C. E. Scheidegger, C. T. Silva, and H. T. Vo. 2005. VisTrails: enabling interactive multiple-view visualizations. In VIS 05. IEEE Visualization, 2005. 135--142.","journal-title":"IEEE Visualization"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"volume-title":"2008 IEEE Symposium on Visual Analytics Science and Technology. IEEE, 99--106","author":"Bier Eric A.","key":"e_1_2_1_4_1","unstructured":"Eric A. Bier, Stuart K. Card, and John W. Bodnar. 2008. Entity-based collaboration tools for intelligence analysis. In 2008 IEEE Symposium on Visual Analytics Science and Technology. IEEE, 99--106."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2013.07.004"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2835202"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858178"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2874313"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858411"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1502650.1502668"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518704"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings, 107--114","author":"Chung Haeyong","year":"2010","unstructured":"Haeyong Chung, Seungwon Yang, Naveed Massjouni, Christopher Andrews, Rahul Kanna, and Chris North. 2010 VizCept: Supporting synchronous collaboration for constructing visualizations in intelligence analysis . VAST 10 - IEEE Conference on Visual Analytics Science and Technology 2010, Proceedings, 107--114."},{"key":"e_1_2_1_14_1","volume-title":"Intelligence analysis: a target-centric approach","author":"Clark Robert M","unstructured":"Robert M Clark. 2013. Intelligence analysis: a target-centric approach 4th ed.). CQ Press, Thousand Oaks, Calif.","edition":"4"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.195"},{"key":"e_1_2_1_17_1","unstructured":"doggyxp. 2018. Braingle: 'The Case of the Mischief Maker' Brain Teaser. (2018). http:\/\/www.braingle.com\/brainteasers\/17325\/the-case-of-the-mischief-maker.html"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sbspro.2013.05.053"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207741"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207711"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.socnet.2018.01.005"},{"key":"e_1_2_1_23_1","first-page":"20","article-title":"Patty Wagstaff's Second Act","volume":"26","author":"Gary D","year":"2011","unstructured":"D Gary. 2011. Patty Wagstaff's Second Act. Air & Space Smithsonian, Vol. 26, 3, 20--25.","journal-title":"Air & Space Smithsonian"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.65.5.959"},{"volume-title":"Interactive Visual Synthesis of Analytic Knowledge. In 2006 IEEE Symposium On Visual Analytics Science And Technology. 51--58","author":"Gotz D.","key":"e_1_2_1_25_1","unstructured":"D. Gotz, M. X. Zhou, and V. Aggarwal. 2006. Interactive Visual Synthesis of Analytic Knowledge. In 2006 IEEE Symposium On Visual Analytics Science And Technology. 51--58."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820071"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481376"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702395"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858364"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240781"},{"key":"e_1_2_1_31_1","volume-title":"Heuer and Center for the Study of Intelligence (U.S.)","author":"Richards","year":"1999","unstructured":"Richards J. Heuer and Center for the Study of Intelligence (U.S.). 1999. Psychology of intelligence analysis. Lulu. com. 184 pages."},{"volume-title":"the art and science of the process of intelligence analysis-preparing for the future of intelligence analysis","author":"Hughes F","key":"e_1_2_1_32_1","unstructured":"F Hughes and D Schum. 2003. Discovery-proof-choice, the art and science of the process of intelligence analysis-preparing for the future of intelligence analysis. Washington, DC: Joint Military Intelligence College."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998196"},{"key":"e_1_2_1_34_1","volume-title":"CrowdWeaver. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work - CSCW '12. ACM Press","author":"Kittur Aniket","year":"2012","unstructured":"Aniket Kittur, Susheel Khamkar, Paul Andr\u00e9 , and Robert Kraut. 2012. CrowdWeaver. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work - CSCW '12. ACM Press, New York, New York, USA, 1033."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531644"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481415"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1979742.1979902"},{"volume-title":"Situated learning: Legitimate peripheral participation","author":"Lave Jean","key":"e_1_2_1_38_1","unstructured":"Jean Lave and Etienne Wenger. 1991. Situated learning: Legitimate peripheral participation. Cambridge university press."},{"key":"e_1_2_1_39_1","volume-title":"Crowdsourcing Intelligence Analysis with Context Slices. Chi '18 Sensemaking Workshop.","author":"Li Tianyi","year":"2018","unstructured":"Tianyi Li, Asmita Shah, Kurt Luther, and Chris North. 2018. Crowdsourcing Intelligence Analysis with Context Slices. Chi '18 Sensemaking Workshop."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837907"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518741"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675283"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998343"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000044"},{"key":"e_1_2_1_45_1","first-page":"341","article-title":"Digilantism: An Analysis of Crowdsourcing and the Boston Marathon Bombings","volume":"57","author":"Nhan Johnny","year":"2017","unstructured":"Johnny Nhan, Laura Huey, and Ryan Broll. 2017. Digilantism: An Analysis of Crowdsourcing and the Boston Marathon Bombings. The British Journal of Criminology, Vol. 57, 2 (2017), 341--361.","journal-title":"The British Journal of Criminology"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025769"},{"key":"e_1_2_1_47_1","unstructured":"Devi Parikh and C. Lawrence Zitnick. 2011. Human-Debugging of Machines. In Neural Information Processing Systems. 1--5. https:\/\/filebox.ece.vt.edu\/ parikh\/human_debugging\/"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of International Conference on Intelligence Analysis","volume":"2005","author":"Pirolli Peter","year":"2005","unstructured":"Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis . Proceedings of International Conference on Intelligence Analysis, Vol. 2005, 2--4. arxiv: gr-qc\/9809069v1"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.2008.1450450234"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.2009.1450460219"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007562322031"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1466620.1466622"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.48.6.1467"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467813"},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"Yla Tausczik and Mark Boons. 2018. Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference on Web and Social Media. https:\/\/aaai.org\/ocs\/index.php\/ICWSM\/ICWSM18\/paper\/view\/17817","DOI":"10.1609\/icwsm.v12i1.15005"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/2772879.2773254"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_2_1_58_1","volume-title":"Context Trees: Crowdsourcing Global Understanding from Local Views . Hcomp 2014 1351131, 210--219","author":"Verroios Vasilis","year":"2014","unstructured":"Vasilis Verroios and Michael S Bernstein. 2014. Context Trees: Crowdsourcing Global Understanding from Local Views . Hcomp 2014 1351131, 210--219. http:\/\/ilpubs.stanford.edu:8090\/1105\/"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143967"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","unstructured":"Jiannan Wang Tim Kraska Michael J. Franklin and Jianhua Feng. 2012. CrowdER: Crowdsourcing Entity Resolution. 1483--1494. 10.14778\/2350229.2350263","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_61_1","first-page":"1","article-title":"Developing a Model of Distributed Sensemaking","volume":"3","author":"Wheat Ashley","year":"2016","unstructured":"Ashley Wheat, Simon Attfield, and Bob Fields. 2016. Developing a Model of Distributed Sensemaking: A Case Study of Military Analysis. Informatics, Vol. 3, 1, 1.","journal-title":"A Case Study of Military Analysis. Informatics"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979407"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124890"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2331791.2331794"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531604"},{"volume-title":"Artificial intelligence, evolutionary computing and metaheuristics","author":"Yampolskiy Roman V","key":"e_1_2_1_67_1","unstructured":"Roman V Yampolskiy. 2013. Turing test as a defining feature of AI-completeness. In Artificial intelligence, evolutionary computing and metaheuristics. Springer, 3--17."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"},{"key":"e_1_2_1_69_1","first-page":"1","article-title":"Supporting Handoff in Asynchronous Collaborative Sensemaking Using Knowledge-Transfer Graphs","volume":"24","author":"Zhao Jian","year":"2017","unstructured":"Jian Zhao, Michael Glueck, Petra Isenberg, Fanny Chevalier, and Azam Khan. 2017. Supporting Handoff in Asynchronous Collaborative Sensemaking Using Knowledge-Transfer Graphs. In IEEE Transactions on Visualization and Computer Graphics, Vol. 24. 1--1.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274374","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274374","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274374","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:35Z","timestamp":1750207475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274374"}},"subtitle":["Solving Mysteries with Crowdsourced Sensemaking"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":69,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274374"],"URL":"https:\/\/doi.org\/10.1145\/3274374","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}